How is text split into tokens?

The most common method for splitting text is byte pair encoding, or BPE. The idea is older than modern AI; it began as a way to compress data. Training a BPE tokenizer starts from a base vocabulary of the individual characters in a body of text. The algorithm then adds new tokens by “learning merges, which are rules to merge two elements of the existing vocabulary together,” repeatedly searching for “the most frequent pair of existing tokens” until the vocabulary reaches its target size.^[1]

Each merge is a rule. Early on the rules join two characters; as training continues, the merged pieces grow into longer subwords.^[1] Common strings like “the” or “ing” end up as single tokens because they appear so often. Rare strings stay broken into smaller pieces. Once the vocabulary is fixed, new text is tokenized by applying those same learned merge rules in order.^[1]

The Hugging Face course walks through a worked example on a small corpus: the pair (“u”, “g”) gets merged into “ug” first, because it shows up 20 times across words like “hug,” “pug,” and “hugs”; the pair (“u”, “n”) merges next, at 16 occurrences; then (“h”, “ug”) merges into the three-letter token “hug,” at 15 occurrences.^[1] Notice the order follows frequency exactly, most common pair first, and that later merges build on earlier ones: “hug” only becomes a candidate once “ug” already exists as a token. Run that same process thousands of times and you get a working vocabulary.

Two properties make this practical. BPE “is reversible and lossless, so you can convert tokens back into the original text,” and it “works on arbitrary text, even text that is not in the tokeniser’s training data.”^[2] It also “compresses the text: the token sequence is shorter than the bytes corresponding to the original text.”^[2] That compression is part of why token counts run lower than character counts.