How do embeddings move from single words to whole sentences?

Early embeddings were per-word. Real applications need a vector for a whole sentence or passage. Sentence-BERT adapted BERT for exactly this, using “siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity,” and the speedup was dramatic: finding the most similar pair in a collection of 10,000 sentences dropped “from 65 hours with BERT / RoBERTa to about 5 seconds.”^[1]

Before Sentence-BERT, a common shortcut was averaging a sentence’s word vectors together into one. That throws away word order entirely: “dog bites man” and “man bites dog” average to the identical vector even though they mean opposite things, since addition does not care what order the terms arrived in. It also blurs distinct meanings together rather than composing them the way a reader actually does.

Sentence-BERT fixes this by training for the sentence-level task directly. Two copies of the same network process a pair of sentences independently (the “siamese” part of the name), each output is reduced to one fixed-length vector by mean-pooling across all of its token embeddings, and the network trains against a loss that pulls semantically similar pairs together and pushes dissimilar ones apart.^[1] The result is a vector built to be compared directly, instead of one that happens to fall out of a per-word representation never designed for that job.

The paper also tested pooling by a single designated summary token (BERT’s own CLS token) and by taking the maximum value per dimension across the sentence, but settled on mean-pooling across every token as the default — their ablation found it outperformed both alternatives, not just that it was simpler to implement.^[1] Averaging in every token’s contribution, rather than compressing the whole sentence through one token’s representation, is what carried over into the models built on this approach since.