Why use tokens instead of whole words?

Why not just give the model a dictionary and treat each word as one unit? Because human language has far too many words, and new ones appear all the time: names, slang, product codes, typos, and other languages. A fixed word list would always be missing something, and any missing word would have to be replaced with a generic “unknown” placeholder, losing its meaning.

Subword tokens avoid this trap. Because the vocabulary includes short pieces, a model can spell out a word it has never seen from familiar parts. Anthropic describes the trade-off directly: “Larger tokens enable data efficiency… while smaller tokens allow a model to handle uncommon or never-before-seen words.”^[2] Common words stay whole and efficient; unusual ones are assembled from their pieces.

Many models push this further with byte-level BPE, which treats text as raw bytes rather than letters. That keeps a small base vocabulary while ensuring “every character you can think of will still be included and not end up being converted to the unknown token.”^[1] The payoff is coverage: with a manageable vocabulary, the model can represent essentially any string, in any language or script, without ever hitting a wall.

GPT-2 shows how small that base vocabulary can be: 256 entries, one for every possible byte, before a single merge has happened.^[1] Every merge after that follows the same simple rule: find whichever pair of adjacent pieces shows up most often across the training text, and fuse it into one new piece. Common pairs merge early — “t” and “h” becoming “th” long before anything else happens to them — while rare combinations stay split far longer, sometimes all the way down to individual bytes, simply because they never come up often enough to earn a merge. Every merge rule learned after that only builds pieces on top of a set that already covers every character there is, which is the real reason byte-level BPE never needs an “unknown” fallback for text it wasn’t trained on. A word list has to grow to keep up with the language; a byte-level vocabulary starts complete.