Why do tokens need to look at each other?

Meaning is contextual. Resolving a pronoun, tracking subject-verb agreement, or disambiguating a word all need information from elsewhere in the sentence. Self-attention is, in the paper’s definition, “an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence.”^[1] Because every token attends to every other directly, distant words influence each other in a single step.

The canonical example comes from the transformer’s own authors: in “the animal didn’t cross the street because it was tired,” the word “it” should attend to the noun it refers to, and attention learns to route that connection.^[2]

An RNN handling the same sentence has no such shortcut. To connect “it” back to “animal,” the information has to survive every intervening hidden-state update between the two words, compressed and recompressed at each step, and the more words in between, the more chances that signal has to fade or get overwritten by whatever else the network is tracking. Self-attention skips all of that: “it” forms a direct connection to “animal” in a single computation, regardless of how many words sit between them. Architecturally, a dependency ten words away costs exactly the same to resolve as a dependency one word away.

Mechanically, that single computation is a matrix multiplication across the whole sequence at once, not a loop that visits each token in turn. Every token’s query is compared against every other token’s key in one matrix multiply, producing a full grid of relevance scores in a single pass; a softmax turns each token’s row of that grid into weights, and a second matrix multiply blends every token’s value vector according to those weights.^[1] Nothing about that computation cares whether the relevant token is next door or a thousand positions away, because the whole grid is produced at once rather than being assembled step by step.

That “exactly the same” claim is about capacity, not guaranteed behavior. In practice, trained models still favor some positions over others, which is part of why very long contexts can still lose track of something buried in the middle even though nothing in the attention mechanism itself makes distance expensive (see long context, lost middle).