How do embeddings power semantic search and RAG?

This is where embeddings earn their keep. Semantic search embeds both the query and the documents and ranks by vector similarity, so a search matches meaning rather than keywords. Cohere frames the contrast directly: semantic search “solves the problem faced by the more traditional approach of lexical search, which is great at finding keyword matches, but struggles to capture the context or meaning of a piece of text.”^[1] It lets “how do I reset my password” match a page titled “recovering account access.”

The same machinery underpins retrieval-augmented generation. The RAG paper paired a generative model with “a dense vector index of Wikipedia, accessed with a pre-trained neural retriever,” fetching relevant passages and conditioning the answer on them.^[2] Against models that only had their frozen training weights to draw on, the paired setup set a new state of the art on three open-domain question-answering benchmarks and produced language its own authors rated more specific, more diverse, and more factual.^[2] The RAG and retrieval page walks through that pipeline end to end.

The mechanism is the same math as the rest of this topic, just run at query time. The user’s question is embedded with the same model used to embed the documents, compared against a pre-built index of document-chunk vectors by cosine similarity, and the top-scoring chunks are pulled out and dropped straight into the model’s context window alongside the original question. The model never retrained on those documents; it is reading them fresh, in the same turn, the same way it reads anything else pasted in.

That dependency is also RAG’s sharpest failure mode. Retrieval sets a ceiling on the answer: if the embedding model does not rank the one genuinely relevant chunk inside the top results, the generator never sees it, and it will still write a fluent, confident answer from whatever it was actually handed. A retrieval miss does not read like a retrieval miss; it reads exactly like a normal, wrong answer.