When does step-by-step reasoning actually help?

Step-by-step reasoning is not always worth it. It earns its keep on problems that genuinely take several steps, and it mostly wastes effort on problems that do not.

The original chain-of-thought paper tested it on arithmetic, commonsense, and symbolic reasoning, the kinds of tasks where a person would also need to work through a few stages.^[1] Anthropic points to the same fit: thinking improves performance on complex tasks like math, coding, analysis, and long-running agentic work, where the quality of the answer depends on intermediate steps that a single-pass response would otherwise compress or skip.^[2] OpenAI describes its reasoning models as suited to “complex problem solving, coding, scientific reasoning, and multi-step agentic workflows.”^[3] The common thread is that the answer cannot be reached in one jump.

The cost is real. Reasoning uses extra tokens and adds time, which means more money and a slower reply. On a simple lookup, a single-fact question, or a short rewrite, that spending buys nothing because there are no intermediate steps to take. For those, a direct answer is both faster and cheaper.

Model size is a criterion too, not just task type. The original chain-of-thought paper found the technique is an emergent ability of scale: it does not help small models, and pays off reliably only around 100 billion parameters and up. Below roughly 10 billion parameters it can actively hurt, because smaller models tend to produce reasoning that reads fluently but follows a broken chain of logic, landing on a worse answer than a direct response would have.^[1] Asking a small model you run yourself to “think step by step” is not the same free trial it is with a large cloud model. It can make the answer worse, not just slower.

Some newer models try to make this choice for you. Anthropic’s later models added adaptive thinking, where the model decides how much to think based on how hard the question looks, rather than thinking the same fixed amount every time.^[2]