What is multi-head attention?

Rather than attend once, the model runs several attention computations in parallel. “Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions,” the paper notes, adding that “with a single attention head, averaging inhibits this.”^[1] The original model used eight heads. Different heads can specialize, one tracking grammar, another tracking which noun a pronoun points to, and their outputs are combined into a richer representation than any single pattern could give.

The averaging problem is concrete, not abstract. Picture a single head trying to track which word a pronoun refers to and which words form the subject and verb of a clause at the same time. With one attention pattern per token, it has to blend both signals into a single weighted average, diluting each relative to a head that only had to track one. Separate heads sidestep this because each computes its own independent query-key-value split and its own independent weighting, so one head is free to specialize on coreference while another specializes on syntax, with neither diluting the other.

The heads are not left as eight separate outputs, either. Their results are concatenated back into one vector and passed through a final learned projection, so the model still produces a single representation per token; the heads only shape how that representation gets built. Scale changes the head count a great deal: GPT-3’s largest version uses 96 attention heads in every layer, twelve times the original count.^[2]

The dimension math is what keeps this affordable rather than eight times the cost. The original model’s total representation size, d_model, was 512; split across eight heads, each one works in its own 64-dimensional subspace rather than the full 512.^[1] Every head still runs the complete attention computation, query against key against value, just in a smaller space, and the eight smaller computations run in parallel for close to the same total cost as one head operating at full width. That is also why head count and model width move together as models scale: GPT-3’s 96 heads only make sense alongside a much larger d_model to divide between them.