What made GPT-3 a turning point?

In May 2020, OpenAI released a paper called Language Models are Few-Shot Learners, describing GPT-3, a model with 175 billion parameters. That was about ten times larger than any comparable model before it.^[1] Underneath, it was a 96-layer decoder-only transformer with a 2,048-token context window, small by today’s standards but the largest anyone had trained at the time.^[2] The training text was weighted rather than pooled evenly: a filtered slice of Common Crawl supplied 60% of it by weight, with the rest split across an expanded web-text collection, two book collections, and Wikipedia, for several hundred billion tokens total.^[2]

The headline was not just the size. Before GPT-3, using a model for a new task usually meant fine-tuning it, which is extra training on labeled examples for that specific job. GPT-3 showed you could often skip that step. You write a few examples of the task directly into the prompt, and the model picks up the pattern and continues it. The paper applied the model to many tasks this way, “without any gradient updates or fine-tuning,” specifying the task purely through text.^[1]

This is called few-shot learning in context, sometimes shortened to in-context learning. One model, frozen after training, could be steered toward translation, question answering, or simple arithmetic just by how you wrote the prompt. That changed the relationship between people and models. Instead of training a new model per task, you described the task in words. The trade only made economic sense because the training run itself was so costly in the first place, one widely cited estimate puts a single pass at around $4.6 million in compute, so spreading that cost across many tasks instead of running a separate fine-tuning job for each was the whole point.^[2] It is the habit that the chat products built on a couple of years later would make ordinary.