How do embeddings make RAG retrieval work?

The machinery that makes retrieval work is embeddings: turning text into vectors so that passages with similar meaning land near each other in space. “Near” is usually measured with cosine similarity, the angle between two vectors, so length does not matter, only direction. You must embed your documents and your query with the same model for the comparison to mean anything — a query embedded by one model and compared against chunks embedded by another is comparing two unrelated coordinate systems that happen to have the same number of dimensions.

How well an embedding model captures meaning in the first place is measured by benchmarks like MTEB, the Massive Text Embedding Benchmark, which scores models across dozens of retrieval, classification, and clustering tasks rather than trusting a single number.^[1] Retrieval then becomes a nearest-neighbor search: embed the query, and find the top-K chunks whose vectors sit closest to it. K is a knob, not a constant — too small and the right passage might not make the cut; too large and you pad the prompt with marginal matches that dilute what the model pays attention to.

The same-model rule fails silently, which is what makes it worth spelling out. Two models can output vectors of identical dimensionality — both 768 numbers long, say — and a nearest-neighbor search between them will run without complaint, returning some result every time. What’s wrong is that each model assigns its own meaning to those 768 axes during training, so a coordinate that captures one idea in one model’s space captures something arbitrary and unrelated in the other’s. Nothing crashes; the search just quietly returns chunks that are not actually close in meaning, which is a harder failure to catch than an error, because nothing signals that retrieval broke.

The Language Model Architecture subject goes deeper on how this representation works; vector databases covers where the vectors actually live once you have more than a few thousand of them.