Why do RAG systems fail despite grounding?

The promise of RAG is grounding: an answer anchored to retrieved, citable text hallucinates less than one pulled from memory alone. But the promise has a condition. Grounding only helps if retrieval surfaces the passage that actually contains the answer. When the chunks are wrong, the model still answers fluently — it just answers fluently from the wrong source, which is a harder failure to catch than an obvious hallucination, because the sentence reads as if it were properly sourced.

In practice, most RAG problems trace back to retrieval, not to the model:

• Chunk size is off. Split too fine and the answer is spread across two chunks, neither of which scores well alone. Split too coarse and the one relevant sentence is diluted by three paragraphs of unrelated text, weakening its embedding.
• The embedding match is weak. A query phrased differently from the source text — different words for the same idea — can land far enough away in vector space to miss, even though a person would immediately see the connection.
• The right passage never makes the top-K. It exists in the index, it is even reasonably close, but three other chunks score marginally higher and the one that mattered gets cut before the model ever sees it.
• The index has gone stale. Documents change; the index only reflects them if something re-embeds and re-indexes after every update. An index that lags the source data will confidently retrieve the old version.

None of these are model failures, and no amount of prompting fixes them, because the model never sees the passage that was never retrieved. That is also why the storage layer matters. Independent comparisons of vector databases note that a handful of options cover most needs; Firecrawl’s review concludes that Pinecone, Weaviate, and Qdrant are the strongest choices for most RAG workloads.^[1] See vector databases for what that storage and search layer actually does.