Tokens & tokenization
Why do tokens matter for cost and context?
You are billed per token and the context window is measured in tokens, so token count drives both cost and how much text fits. Token counts also differ between models, so the same text can cost more on a different tokenizer.
Tokens are the unit you pay for. Model pricing is quoted per million tokens, and input and output are priced separately. Anthropic’s published rates, for example, charge a lower price for the tokens you send in than for the tokens the model writes back.[1] A handy estimate from the same pricing page: “1 token is approximately 4 characters or 0.75 words in English.”[1]
Anthropic’s own worked example makes the arithmetic concrete: a one-hour session on Claude Opus 5 that sends 50,000 input tokens and gets back 15,000 output tokens totals $0.705, with the 15,000 output tokens alone accounting for $0.375 of that.[1] Turn on prompt caching so 40,000 of those input tokens are cache reads instead of fresh ones, and the same session drops to $0.525, because a cache hit is billed at a tenth of the standard input price.[1]
Tokens are also the unit of a model’s working memory. The context window “refers to all the text a language model can reference when generating a response, including the response itself,”[2] and it is measured in tokens. Everything has to fit: your prompt, any documents, the conversation so far, and the reply. Run out of room and older content has to be dropped or summarized. So token count, not word count, sets both your bill and your limits.
Size alone doesn’t change the per-token rate, though. Anthropic bills a 900,000-token request at the same per-token price as a 9,000-token one on models with the full context window included.[1] The bill scales with how much you actually send, not with the size of the window you have access to.
One catch worth knowing: token counts are not universal. Different models use different tokenizers, so the same text can come out to a different number of tokens. Anthropic notes that “Claude 4.7 and later models… use a newer tokenizer” that “produces approximately 30% more tokens for the same text” than the one earlier models used.[1] A document that costs a certain number of tokens on one Claude generation is not a safe estimate for the next one. When budgeting, count tokens with the tokenizer for the model you are actually using.
Counting tokens
- tiktoken ↗
OpenAI's open BPE tokenizer; counts tokens for OpenAI models so you can estimate length and cost.
- Anthropic token counting ↗
API that returns the token count for a request before you send it to Claude.
References
- Pricing — Anthropic
- Context windows — Anthropic
- tiktoken README — OpenAI