Where did the training data for the deep-learning takeoff come from?

A learning method is only as good as what it can learn from. Before 2012, most labeled image collections held tens of thousands of pictures, which was too few to train a large network without it simply memorizing. The standard research benchmark at the time was PASCAL VOC, established in 2005 with about 20,000 images spread across twenty object categories.^[3] The dataset that changed this was ImageNet, introduced in 2009, and by the time of its 1000-category competition subset it ran roughly fifty times the categories and sixty times the images of what researchers had been training on before.^[3] The paper describes more than 3.2 million high-resolution images organized by the nouns of WordNet, a lexical database that arranges concepts into a hierarchy.^[1] Human workers labeled the images, so each one came with a trustworthy category.

Getting millions of images labeled by hand was itself the hard problem. Fei-Fei Li’s team solved it by crowdsourcing the work through Amazon Mechanical Turk, where “49K workers from 167 countries” filtered and labeled candidate images between July 2008 and April 2010.^[3] Each of the roughly 14 million images that resulted was labeled three separate times to catch mistakes, and by 2012 ImageNet had become “the world’s largest academic user of Mechanical Turk.”^[3] No single research lab could have hand-labeled a set that size on its own; paying a global crowd small amounts per image is what made a dataset like this possible at all.

ImageNet kept growing, and the AlexNet paper describes a collection of over 15 million labeled images in roughly 22,000 categories.^[2] Training and evaluation used a standardized subset from the annual ImageNet competition: roughly 1.2 million training images, 50,000 for validation, and 150,000 for testing, across 1000 categories.^[2]

This scale is what let a 60-million-parameter network learn without badly overfitting. A small dataset would have let the network memorize answers; a large and varied one forces it to find patterns that hold up on images it has never seen.