What is a token?

A language model does not read text the way you do. Before it sees your sentence, the text is chopped into small pieces called tokens. A token is the basic unit the model works with. As Anthropic’s documentation puts it, tokens “are the smallest individual units of a language model, and can correspond to words, subwords, characters, or even bytes.”^[1]

So a token is sometimes a whole word, but often it is a fragment. The word “cat” might be one token, while “tokenization” might be split into several. Spaces and punctuation count too. A rough guide for English is that one token stands for about 3.5 characters, though the exact number changes with the language being used.^[1] Zoomed out to whole words instead of characters, a common rule of thumb puts it at roughly 100 tokens per 75 English words — close enough for a quick estimate, but not interchangeable with a word count for anything that actually bills by the token.

You normally never see tokens. They are hidden when you type a prompt and read a reply. They become “relevant when examining the exact inputs and outputs of a language model,”^[1] which is to say when you care about how long your text is, how much it costs, or how much will fit.

The choice of tokenizer is not a cosmetic detail. Anthropic’s own glossary notes that “the choice of tokenization method can impact the model’s performance, vocabulary size, and ability to handle out-of-vocabulary words,”^[1] and that the 3.5-characters-per-token estimate above “can vary depending on the language used.”^[1] Two models reading the identical sentence can walk away with a different number of tokens, simply because they learned different pieces to split it into. The next pages cover how text gets turned into tokens, and why the model uses these pieces instead of plain words.