What came before the transformer, and why was it slow?

The transformer is the neural network architecture behind essentially every modern large language model. It arrived in the 2017 paper “Attention Is All You Need,” written by an eight-person team mostly at Google, proposing a design “based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.”^[1]

Before transformers, the standard way to process language was the recurrent neural network (RNN), which reads a sentence one word at a time, carrying a running memory forward. That design has two problems. It is slow to train, because each step has to wait for the one before it. And it tends to forget: information about early words weakens as it passes through many steps. The transformer’s answer is to drop recurrence and let every word look at every other word directly, in a single parallel operation.^[2]

The bottleneck was not just conceptual, it was computational. A GPU earns its speed by running huge numbers of independent multiplications at once, but an RNN’s recurrence forces step t to wait on the hidden state from step t-1, so the sequence has to be walked one position at a time no matter how many cores sit idle. Self-attention turns “how does token A relate to token B” into a single matrix multiplication computed for every pair of positions at once, which is exactly the shape of computation a GPU is built for.

The payoff was concrete. Training on eight P100 GPUs, the base model needed just 12 hours (100,000 steps) to beat the previous best English-to-German translation result, including ensembles of multiple models, by more than 2 BLEU points; the larger model needed 3.5 days to set a new single-model state-of-the-art on English-to-French, at what the authors call “a small fraction of the training costs of the best models from the literature.”^[1] Gated designs like the LSTM had already made RNNs better at retaining earlier words, but gating did nothing for the sequential bottleneck itself. Attention had even been grafted onto an RNN encoder-decoder three years earlier to fix a different problem (see where attention came from) — the RNN was still there, still stepping through the sequence one word at a time. Vaswani et al.’s move was removing the recurrence entirely, not patching around it.