How do you pick an embedding model?

Embeddings differ in quality by task and by language, so they are benchmarked. The Massive Text Embedding Benchmark (MTEB) spans many task types and languages, and its headline finding is a useful warning against shopping for a single winner: “no particular text embedding method dominates across all tasks.”^[1] The right embedding model depends on what you are doing with it.

The reason no model dominates is straightforward: embedding models are trained on different data mixtures for different objectives, so a model tuned on broad web text can lag behind a model tuned specifically for code, for a non-English language, or for matching a short query against a long document (asymmetric search) rather than comparing two similar-length passages (symmetric search). MTEB reports results broken out by task category, classification, clustering, retrieval, semantic similarity, precisely because a model that tops one category regularly lands mid-pack on another.^[1]

Dimensionality is part of the same tradeoff, and it is not free in either direction. A higher-dimensional embedding can encode finer distinctions, but every extra dimension adds storage and compute cost across an index that might hold millions of vectors, so the model at the top of a leaderboard is not automatically the right one to put into production at scale. A model scoring a point or two higher on MTEB’s average is rarely worth doubling a storage bill if the cheaper, smaller model already clears the bar the actual task needs; the benchmark measures average capability across many tasks, not cost per correct match on yours.

That is also why the average score is the wrong number to lead with. A customer-support search tool lives or dies on the retrieval and semantic-similarity categories, not clustering or classification, so filtering the leaderboard down to those categories before comparing models gets closer to a real answer than reading the single headline average ever will.