What is chain-of-thought prompting?

A chain of thought is the model writing out its working before it commits to an answer. Instead of jumping straight to “42”, it produces a series of short intermediate steps, the way a person might think aloud through a word problem, and only then states the result.

The idea comes from a 2022 paper by Jason Wei and colleagues at Google, which defined a chain of thought as “a series of intermediate reasoning steps” and showed that prompting a model to generate one “significantly improves the ability of large language models to perform complex reasoning.”^[1] Their method was to put a few worked examples in the prompt where each example included the reasoning, not just the answer. The model then copies that pattern: it reasons step by step on your new question too.

The gains were large on problems that need several steps. Prompting a 540-billion-parameter model with just eight chain-of-thought examples reached the best published accuracy at the time on GSM8K, a set of grade-school math word problems, beating a fine-tuned model paired with a separate answer-checker.^[1]

Why does showing the work help? A model generates its answer one token at a time, and each token it writes becomes part of what it reads next. Writing the steps gives it more room to break a hard question into smaller pieces it can handle, instead of forcing the whole answer out in a single leap.

There is a sharper way to state that. Every token a model produces gets roughly the same fixed amount of computation, whether that token is a single digit or the hardest part of the problem. A direct answer has to fit all of a multi-step problem’s reasoning into whatever budget a handful of output tokens can carry. Writing out the intermediate steps does not give any single token more computation to work with; it gives the problem more tokens to spread that work across, trading length for total thinking rather than depth per step. The next page covers a shortcut version that needs no examples at all.