How did BERT and GPT build on the Transformer?

The Transformer was a design, not a finished product. In 2018 two papers showed what to do with it, and they pointed in different directions. Both shared one move: train the model on a large pile of plain, unlabeled text first, then adjust it for the job you actually want.

BERT, from a Google team, stands for Bidirectional Encoder Representations from Transformers. It reads a sentence in both directions at once, “by jointly conditioning on both left and right context in all layers.”^[1] It learns by playing a fill-in-the-blank game: hide some words and make the model guess them from what surrounds them, specifically 15% of the input tokens on any given pass, plus a second task where it has to decide whether one sentence plausibly follows another.^[3] The paper released two sizes, a 110-million-parameter BERT-base and a 340-million-parameter BERT-large, both pre-trained on the same two text collections: Toronto’s BookCorpus, about 800 million words, and English Wikipedia, about 2,500 million words.^[3] Once pre-trained, BERT “can be fine-tuned with just one additional output layer” for tasks like question answering, and it pushed the GLUE benchmark score to 80.5%, a 7.7 point jump.^[1] BERT is built to understand text.

GPT, from OpenAI, takes the other path. Its paper, “Improving Language Understanding by Generative Pre-Training,” describes “generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task.”^[2] It reads left to right and predicts the next word, which makes it good at producing text. The original model was small by later standards, 117 million parameters in a twelve-layer decoder-only transformer.^[4] Its training text was BookCorpus: roughly 7,000 self-published novels, romance and science fiction and fantasy, scraped from the indie ebook site Smashwords, the same collection that fed BERT’s pre-training alongside Wikipedia.^[5] The paper reported improving the state of the art on 9 of the 12 tasks it studied.^[2] A line of models that would go on to hold real conversations first learned to write from genre paperbacks nobody outside that community had heard of.