What did the 2017 transformer paper actually introduce?

In June 2017, a team of researchers published a paper with a memorable title: “Attention Is All You Need.”^[1] It introduced a network design they called the Transformer. The idea that made it new was in the title. Earlier language models processed text by reading it in order, one step feeding the next. The Transformer threw that out and was, in the authors’ words, “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”^[1]

Attention is a way for the model to look at every word in a sentence at once and decide which other words matter for understanding each one. Because it does not have to step through text in sequence, the work can be split across many processors at the same time. The authors reported that their models were “more parallelizable and requiring significantly less time to train.”^[1] Under the hood, each word gets turned into three vectors, a query, a key, and a value, and attention scores come from comparing queries against keys.^[2] The paper runs several of these comparisons side by side as “multi-head attention,” so “each attention head learns different linear projections,” letting the model track several different kinds of relationship between words at once instead of settling for just one.^[2] The base model in the paper used eight of these attention heads across six encoder layers and six decoder layers, about 65 million parameters in total, small enough that the whole thing trained in roughly 100,000 steps.^[1] That is a tiny model by the standards of what came after it, which is part of what made the result so surprising: the architecture, not sheer scale, was doing the work.

The proof was in translation. On a standard English-to-German task the Transformer reached 28.4 BLEU, a common translation quality score, beating the prior best by more than two points. On English-to-French it set a new single-model record of 41.8 BLEU after training for three and a half days on eight GPUs.^[1] Better results and faster training, from a simpler design, was a combination the field had not seen before.