What are a prompt, a token, and the context window?

A prompt is whatever you type in to get a response. Google defines it as “any text entered as input to a large language model to condition the model to behave in a certain way.”^[1] A prompt can be a question, an instruction, an example, or a long document you want summarized. It is the one part of the system you control directly.

A token is the unit the model actually reads in. Text “is first broken into units called tokens, which are words, character sets, or combinations of words and punctuation, by a tokenizer.”^[2] A token is often a piece of a word rather than a whole one. A rough rule of thumb is that one token is about three-quarters of a word in English, so a short paragraph might be a hundred or so tokens. Models count work in tokens, which is also why usage is usually priced per token.

Microsoft’s own worked example shows the mechanics: the sentence “I heard a dog bark loudly at a cat” splits into nine word-tokens, with the repeated “a” reusing the same token ID rather than getting a new one each time.^[2] Token size is a real trade-off, not a free choice. Smaller tokens, down to individual characters, let a model handle typos and unfamiliar words better, but need more of them to cover the same text; larger, whole-word tokens process faster but choke on anything outside the vocabulary.^[2]

The context window is how much the model can hold at once. It is “the maximum amount of text or other tokenized input available to the model at one time when generating output,” and it “is usually measured in tokens.”^[3] Think of it as the model’s working memory: the material it can see while writing a reply. This budget covers both your input and the model’s output together.^[2] Anything that falls outside the window is not available unless it is provided again, summarized, or retrieved.

The tradeoff runs deeper than raw capacity. Even with room to spare, models don’t use a long context evenly: performance on long-context tasks is often worse when the relevant information sits in the middle of the input rather than near the beginning or end, an effect researchers call “lost in the middle.”^[3] By the mid-2020s some systems advertised windows in the millions of tokens, but a bigger window doesn’t by itself guarantee the model uses everything inside it.^[3]