Why do context windows have a limit at all?

The limit comes from how attention works. In a transformer, self-attention has every token compare itself to every other token in the sequence. The original transformer paper gives the cost of a self-attention layer as on the order of n squared times d, where n is the sequence length and d is the representation size.^[1] That n-squared term is the catch: the work grows with the square of the length, not in proportion to it.

In practical terms, doubling the number of tokens in the window roughly quadruples the attention computation, and the memory needed to hold the comparisons rises the same way. A window twice as long is not twice as costly to run; it is closer to four times. This is why model makers set a ceiling on the window rather than allowing any length, and why pushing windows into the hundreds of thousands or millions of tokens has taken real engineering effort.

The same paper notes that self-attention is fast compared to older recurrent layers “when the sequence length n is smaller than the representation dimensionality d, which is most often the case with sentence representations used by state-of-the-art models.”^[1] Once sequences get very long, that advantage narrows and the n-squared cost dominates. Reducing this cost is the focus of a large body of follow-up work on more efficient attention.

In practice, serving systems avoid paying that quadratic cost twice. Once a token’s key and value vectors are computed, they get cached, so producing the next token only requires computing attention for the one new position against the stored cache, not recomputing every pairwise comparison from scratch. Without this KV cache, every new token would cost as much as reprocessing the entire conversation so far; with it, the expensive n-squared pass only happens once, on the first read of a given piece of context. The cache itself still has to be held in memory and grows with context length, a hardware cost this wiki’s running AI yourself section covers in concrete gigabytes.