Where did attention come from?

Attention was not born as the whole architecture. Bahdanau, Cho, and Bengio introduced it in 2014 to fix a specific limitation in translation: an RNN had to compress an entire source sentence into one fixed-length vector. They “conjecture[d] that the use of a fixed-length vector is a bottleneck” and let the model instead “automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word.”^[1] That soft-search is the direct ancestor of modern attention; the explicit query-key-value framing came later, with the transformer.

The mechanism itself worked through a small feedforward network, the “alignment model,” which scored how well each encoder hidden state matched the decoder’s current state; those scores were normalized and used to weight a sum over the encoder states.^[1] That is functionally a query (the decoder’s state), a set of keys (the encoder states, scored for match), and a set of values (the same encoder states, pulled in by weight). The transformer’s vocabulary for this arrived three years later, but the shape of the operation was already there.

The scoring function itself changed along the way, and the change mattered for speed as much as elegance. Bahdanau’s alignment model computed its scores with a small feedforward network, concatenating the decoder state and each encoder state and learning a nonlinear function over that combination — additive attention.^[1] The transformer replaced this with scaled dot-product attention: no feedforward network at all, just a matrix multiplication between queries and keys, divided by the square root of the key dimension to keep the values from growing too large before the softmax.^[2] A dot product is dramatically cheaper to compute in parallel on a GPU than a per-pair feedforward pass, which is a real part of why the newer formulation scaled where the older one struggled to.

What Bahdanau’s design did not do was remove the RNN. The encoder and decoder were both still recurrent networks, stepping through their sequences one position at a time; attention was added on top to fix the fixed-vector bottleneck, not the sequential one. The insight in “Attention Is All You Need” was that the attention operation, on its own, was doing enough of the real work that the RNN underneath it could simply be deleted (see before transformers).