How does a language model actually produce text?

A language model does not write a whole sentence in one shot. It produces text one token at a time, where a token is a chunk of text such as a word or a piece of a word. The model looks at everything written so far, predicts the single most fitting next token, adds that token to the text, and then runs again on the slightly longer text. It keeps repeating this loop until it decides to stop.

This is called autoregressive generation, because each prediction feeds back in as input for the next one. The original transformer paper describes it plainly: “At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next.”^[1] In probability terms, the chance of each token depends only on the tokens before it.^[2]

The important thing to picture is the loop. The model has no separate “planning” stage and no draft it revises. Every word you read was chosen the same way: one prediction at a time, each one informed by all the text that came before.

There is an apparent contradiction worth resolving here: the transformer processes a whole sequence in parallel, so how can generation still happen one token at a time? Training and inference run the loop differently. During training, the model sees a full sequence at once and predicts every position’s next token simultaneously, using a causal mask that blocks each position from seeing anything after it; the ground-truth tokens are fed in regardless of what the model itself would have guessed, a trick called teacher forcing. At inference there is no ground truth to feed in, only the model’s own choices, so it genuinely cannot skip the loop: token 50 cannot be produced until token 49 has actually been chosen, because predicting token 50 depends on token 49 being there to attend to.