Blog · August 2026
Three days ago we published Characters Are Not Tokens, in which we measured thirty-two prompt substitutions and found nine that cost tokens rather than saving them. One of the nine was for example → e.g..
That rule was still running in our own compression engine on the day the post went live.
This is what happened when we stopped measuring substitutions one at a time and measured every rule we ship, against every tokenizer our traffic actually reaches.
OpenAI publishes tiktoken. Meta, Alibaba and DeepSeek ship tokenizer.json files you can load offline. Anthropic publishes neither — there is no BPE to download, which is why most tooling quietly gives up and estimates.
What Anthropic does publish is count_tokens: an endpoint that returns the exact token count for any input, free of charge, and not billed against your token quota. It gives you counts, not pieces — you cannot see where the merges fall, but you can see precisely what they cost. That is enough, with two corrections.
Correcting for the message envelope
A counted message includes chat scaffolding, so count("hello") is not the cost of hello. Solve for the constant by repetition:
E = 2 × count(U) − count(U + U)
Six tokens, on our account, confirmed against a second unit string before we trusted it.
Correcting for the leading space
BPE merges a leading space into the word that follows, so hello mid-sentence is not the same token sequence as hello at the start of one. The in-context form is the one a compression rule actually rewrites. Measure it with a carrier:
n_space(w) = count("x " + w) − count("x")
The carrier means you never send a leading space at all, so nothing upstream can normalise it away before it is counted.
After that it is only volume. We measured 150,895 words — every entry in an English dictionary, plus WordNet's lemmas, plus a few hundred mathematical symbols — in both forms, across seven tokenizers. About 300,000 calls at roughly 240 words a second. Twenty minutes, and it cost nothing.
Average tokens per dictionary word, space-prefixed, across 150,673 English words:
Tokens per word, by tokenizer
Claude spends 3.72 tokens on the average dictionary word where the open BPEs spend 2.3 to 2.5. That single fact reorders the problem in both directions at once.
It means there is more to compress, not less. More words arrive as several tokens, so more of them have a cheaper synonym: 22,395 words have a same-meaning substitute that costs fewer tokens on Claude, against roughly 18,000 on every other tokenizer we measured. The densest tokenizer is the largest opportunity.
It also means rewrites that are free elsewhere are worth real money here. demonstrate → show is five tokens down to one on Claude, and one down to one on GPT-4o — a rule that looks pointless if you only ever test it against tiktoken.
And it means the reverse holds too.
Our engine had a rule that rewrote should not to shouldn't. One character shorter, and the sort of tidy-up that looks free.
"should not" → "shouldn't", in context
The same rewrite is a win on one tokenizer, neutral on another, and a threefold expansion on the third. The rule was never wrong in the abstract — it was wrong for a third of the text we send. Claude is 21% of our requests and 33% of our prompt tokens: the largest single share of any tokenizer family we serve, and nowhere near a majority. No family holds one, which is precisely why a single global rule set was the wrong shape to begin with.
All sixteen contraction rules cost Claude tokens. should not and would not are three each; could not, does not, was not, has not and had not are two; the rest are one. Every one of them was neutral or positive on the tokenizers we could measure before this month, which is exactly why they survived review.
We extracted every literal-phrase rule the engine ships — 203 of them — and measured the exact in-context delta of each, on all seven tokenizers. Thirty-eight were zero or negative for at least one family.
Sixteen were the contractions. The one that stung was for example → e.g.: Claude 2 tokens to 4, o200k 2 to 3, because e.g. splits at the full stops. We had published that finding on a Sunday and were still shipping the rule on the Wednesday. Measuring a thing and acting on it are separate projects, and only one of them was on a roadmap.
Four things, in the order they had to happen.
The cache key, first and alone. Our compression cache keyed on the text, the level, the model profile and the request policy — but not on the tokenizer family. That was harmless for exactly as long as compression output never differed by family. The moment it did, a Claude request and a GPT request with identical text would have been served each other's cached output. It shipped by itself, as a behaviour-neutral change, before anything that could trigger it.
Suppression rather than deletion. The engine already had a per-family gating mechanism, shipped some weeks ago with a deliberately empty table and a comment setting the evidence bar: an effect that replicates across contexts, not an aggregate on a single corpus. An earlier candidate had been retracted for precisely that mistake. We filled the table with 28 entries and deleted nothing — rule identifiers are positional, so removing one renumbers every rule after it and orphans months of telemetry. Suppressed rules stay where they are, masked per family, each carrying its measured per-tokenizer deltas in a comment beside it.
A practical note from that work: read the rule identifiers from the running engine rather than from the source. Parsing the source file came out off by one, and an off-by-one in a suppression table is a rule you believe you have disabled and have not.
A curated synonym tier. 2,075 pairs cleared the mechanical bar — at least two Claude tokens saved, negative on no tokenizer, and frequent enough in real English to ever fire. We shipped twenty-four of them. The rejections are the more useful half: WordNet will cheerfully report that medicine and music share a synset, and that nothing can be replaced with null. Others break on part of speech (telephone number does not survive telephone → call), or quietly narrow the meaning (nearly is not about), or shift the register in a way no token count will ever flag. The review file is checked in, rejections included, because the next person to extend the list needs those more than the acceptances.
Counting Claude locally. The 150,000-word measurement is now embedded in the engine binary — sorted, binary-searched, about ten megabytes — with a segmentation pass that prices punctuation and digit runs from the same calibration. It lands within 1.1% of count_tokens on realistic prompts, which is close enough to catch a rule that costs tokens and not nearly close enough to be called exact.
So we do not call it exact. It reports through its own counting mode, and the flag that says "these numbers came from a real encoder" stays false, because the guard that decides whether to keep a compression must never mistake a good estimate for a measurement. It ships switched off, and its first live setting is report-only: it will publish the verdict without acting on it. Enforcing it changes behaviour on roughly a third of Claude traffic, and that deserves an observation window rather than a release note.
We ran the previous engine and the new one side by side against the same fifteen Claude-destined prompts, then counted every output with count_tokens.
Same prompts, both engines, exact Claude tokens
The old engine was handing back most of its own savings. On three of the fifteen prompts it made the prompt larger: the worst went from 27 tokens to 34, a compression that cost 26% more than sending the text untouched. The new engine regressed on none of them.
The honest caveat, because this is the kind of number that gets quoted without one: that corpus is deliberately dense in the patterns we changed. On a general prompt corpus the difference is far smaller — mean savings moved from 7.08 to 7.17 tokens per prompt. Real traffic sits somewhere between the two, depending on how contraction-heavy your prompts happen to be. What holds without qualification is the direction: the measured negative rules are gone, and nothing got worse.
Measure in the unit you are billed in. A character-based metric reports a saving for any rewrite that shortens text, which means it can never report the failure that matters. We wrote that three days ago about other people's optimisations; the audit is what it looks like to take your own advice.
A tokenizer you cannot download is not a tokenizer you cannot measure. A counting endpoint and two calibration constants produced a 150,000-word table in twenty minutes for nothing. We had assumed for months that Claude was unmeasurable. It was only unpublished.
When the evidence is new, suppress rather than delete, and ship the enforcement dark. Both cost almost nothing, and both leave a way back. The measurement that overturned sixteen rules this week is exactly the kind that deserves to be reversible next week.
The numbers behind this post
If you want to check the method rather than take our word for it, none of it is proprietary: count_tokens is public, tiktoken is public, and the two corrections above are four lines of arithmetic. The tokenizer is not a mystery. It is just something almost nobody measures before optimising against it — including, for three days in August, us.
No account needed to start chatting — or bring your own API key.