What happened at the 2012 ImageNet competition?

People often point to one event as the start of the modern deep-learning era: the 2012 ImageNet competition. A deep neural network built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton, now usually called AlexNet, won by a wide margin. It scored a top-5 test error rate of 15.3%, compared with 26.2% for the second-best entry.^[1] A “top-5 error” counts an image as wrong only when the correct label is not among the model’s five most probable guesses.^[1]

The size of that gap is what made the result hard to ignore. Most of the strong entries in those years used hand-built feature extractors followed by a classifier. AlexNet instead learned everything from the raw pixels with a single deep network of eight learned layers: five convolutional and three fully connected, ending in a 1000-way output.^[1] The network had about 60 million parameters and 650,000 neurons.^[1]

Depth alone was not the whole story. AlexNet also popularized three smaller ideas that later became standard: the non-saturating ReLU activation function, which “trained better than tanh and sigmoid”; dropout regularization at a 0.5 probability, which randomly switches off neurons during training to keep the network from overfitting; and data augmentation computed on the fly, extracting random 224×224 patches and their mirror images from each training photo, which alone multiplied the effective training set 2048-fold.^[2] None of the three were new inventions, but bundling all of them into one network at this scale was.

The win did not come from one clever trick. It came from three older ideas finally arriving at the same time at usable scale: a large labeled dataset, graphics hardware fast enough to train a big network, and the back-propagation learning method run on a far larger model than before. The rest of this topic takes those one at a time.