Do bigger context windows mean the model uses all of it well?

Windows have grown a lot. Google DeepMind’s Gemini 1.5 report describes models “capable of recalling and reasoning over fine-grained information from millions of tokens of context” and reports “near-perfect recall (>99%) up to at least 10M tokens” on a retrieval test where a single fact is hidden in a very long input.^[1] So a model can, in the right test, find a needle in a haystack of millions of tokens.

A bigger window is not used evenly, though. The study “Lost in the Middle” examined how models handle information placed at different spots in a long input and found that “performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts.”^[2] The authors note this held “even for explicitly long-context models.”^[2] A document can be inside the window and still get less attention because of where it sits.

Anthropic’s documentation makes a related point about quantity: “more context isn’t automatically better,” and “as token count grows, accuracy and recall degrade, a phenomenon known as context rot.”^[3] The practical takeaway is that what you put in the window, and where, matters as much as how much fits. Putting the most important material near the start or end, and trimming what the model does not need, tends to help.

Researchers do not have full agreement on why the effect is shaped this way, but two candidate explanations come up often: positional encodings make it harder for attention to resolve a precise relative distance once two tokens are far apart, and causal masking means early tokens accumulate attention from every token that follows them, while tokens stuck in the middle compete for a limited attention budget from both directions at once. Whatever the exact mechanism, the practical shape of the problem is what a prompt writer has to work around day to day; the prompting section covers that from the writing side, not the architecture side.