Why do AI models make up facts, and how do I reduce it?

A hallucination is when a model states something false as if it were true. The text reads well and sounds confident, which is exactly what makes it dangerous.

The cause is built into how these systems work. A model is trained to produce the most likely continuation of text, not to check facts against the world. One study frames the problem in terms of how models are trained and graded: the procedures reward guessing over admitting uncertainty, so a model behaves “like students facing hard exam questions” and guesses when unsure, “producing plausible yet incorrect statements” instead of saying it does not know.^[1] If a wrong guess scores the same as silence, guessing is the safer bet for the model.

The most direct fix is to remove the penalty for honesty. Give the model explicit permission to say it is unsure. Anthropic’s guidance lists this first: “Allow Claude to say ‘I don’t know’… This simple technique can drastically reduce false information.”^[2] A line like “If the document does not contain the answer, say so” is often enough.

The second fix is grounding. Instead of asking the model to answer from memory, give it the source material and ask it to base its answer on that. For long documents, the same guidance suggests asking the model to pull exact quotes first, then answer using only those quotes, and to drop any claim it cannot support with a quote.^[2] This keeps the answer tied to text you can check.

The two steps work together for a specific reason. Pulling the quote first forces the model to point at real text before committing to a claim, so a wrong answer now has to contradict something sitting in its own context rather than just misremembering a detail from training. Answering from a quote already in front of it is a different kind of task than answering from memory: the material is present and checkable, not recalled and hoped-for. This is also why retrieval helps with hallucination at a larger scale than one pasted document: it feeds the model real source text automatically instead of leaving the answer to whatever it happened to absorb during training.