How did GPUs make training a deep network possible?

A graphics processing unit, or GPU, was built to draw images by doing many simple arithmetic operations at once. It turns out that training a neural network is also mostly many simple operations done at once, so the same hardware fits the job well. By 2012 this match had become the deciding factor.

The architectural reason is straightforward. A CPU has a handful of powerful cores built to run one complicated instruction stream quickly and in order. A GPU flips that trade: thousands of small, simple cores that all run the same basic arithmetic, like the multiply-and-add at the heart of a neural network’s forward and backward pass, on different pieces of data at the same time. Rendering a scene and training a network both boil down to that same shape of work: the same few operations, applied to enormous amounts of data, in parallel.

The AlexNet authors were direct about it. They wrote that convolutional networks had “still been prohibitively expensive to apply in large scale to high-resolution images,” and that current GPUs were “powerful enough to facilitate the training of interestingly-large CNNs.”^[1] They also said the depth of the model was important to its accuracy, and that depth was what the hardware made affordable.^[1]

The concrete numbers are modest by today’s standards. The network trained on two GTX 580 GPUs with 3 GB of memory each, and took between five and six days to finish.^[1] Two GPUs weren’t a convenience, they were a requirement: the full network didn’t fit in one card’s 3 GB of memory, so the authors split it into two near-identical halves, one per GPU, that only exchanged information at a handful of layers and merged back together at the very end.^[2] The authors noted that the network’s size was limited mainly by GPU memory and the training time they were willing to accept, and predicted results would improve with faster GPUs and bigger datasets.^[1] That prediction held up.