What is self-attention, and why is it the transformer's key move?

The mechanism that makes this work is self-attention: for each word, the model weighs every other word by how relevant it is, and blends them in. A word’s meaning can be shaped directly by any other word, no matter how far away. (The attention page covers the mechanism in detail.)

Because there is no left-to-right dependency, the whole sequence is processed at once. That parallelism is the practical breakthrough: it is what let models train fast enough on GPUs to grow to their current size. On the 2014 English-to-German translation benchmark the original transformer beat the best prior recurrent and convolutional models while being, in the authors’ words, “more parallelizable and requiring significantly less time to train.”^[1]

The name is precise: it is attention where a sequence attends to itself, rather than one sequence attending to another. That distinction traces back to how attention began. Bahdanau’s 2014 mechanism let a translation decoder attend to a separate encoder sequence (see where attention came from). The transformer generalized the same operation so a sequence’s own tokens relate to each other directly, which is what makes it useful for understanding a single piece of text, not only for translating between two.

Comparing every token to every other token is not free, and the cost doesn’t grow gently: the number of pairwise comparisons scales with the square of the sequence length, so doubling the context roughly quadruples the work rather than doubling it. The context windows topic covers what that cost means in practice as sequences grow. What self-attention buys in return is depth of relationship. Every block in the stack repeats the same all-pairs comparison (see the block, repeated), so a model does not just relate words once. It re-relates them dozens of times, each pass building on a representation already reshaped by the pass before it.