What is an embedding, and how does it turn meaning into geometry?

An embedding is a list of numbers, a vector, that represents a piece of text as a point in a high-dimensional space, arranged so that the geometry encodes meaning. OpenAI’s definition is about as direct as it gets: “an embedding is a vector (list) of floating point numbers. The distance between two vectors measures their relatedness.”^[1] Items with similar meaning land close together; unrelated items land far apart.

The idea grew out of a simple observation: words used in similar contexts tend to mean similar things, so they should get similar vectors. word2vec showed in 2013 that you could learn such vectors cheaply at scale, and that the resulting space captured real structure, both grammatical and semantic.^[2] The numbers are learned rather than assigned, so each dimension is a latent feature, not a label a human picked.

The geometry is not just proximity; it also encodes relationships as directions. The word2vec paper introduces its most famous example as prior work rather than its own discovery, citing an earlier paper for the finding that “vector(‘King’) - vector(‘Man’) + vector(‘Woman’) results in a vector that is closest to the vector representation of the word Queen,” then builds its own contribution on top of it.^[2] That means a single direction in the space behaves like “male to female,” and applying the same direction elsewhere tends to produce a similar shift, which is a much stronger claim than “similar things sit close together.”

What the word2vec paper itself contributed was turning that one striking example into a measurable, repeatable test. It assembled a battery of analogy questions across categories such as capital cities, currencies, and adjective-to-adverb pairs, plus simpler ones like “biggest is to big as smallest is to small,” and scored how often the nearest vector to the arithmetic actually landed on the intended word.^[2] That turned “the geometry seems to capture meaning” into a number you could improve architecture against, rather than a single anecdote.

The training signal behind this is simple: words that tend to appear in similar surrounding contexts get pushed toward similar vectors, and words that never share context drift apart. Nobody tells the model that “king” and “queen” are related; the relationship falls out of how those words actually get used across enough text. Modern embeddings keep the same idea at industrial scale: OpenAI’s current models represent text as 1,536 or 3,072 numbers per vector, thousands of times more geometry than word2vec’s original space, encoding shades of meaning no one hand-labeled.^[1]