Sixty-Four Rewrites, Zero Tokens Saved

The Porter stemmer turns Andes into and. On a hand-built list of proper nouns, 41 of 51 tokens came back damaged.

Everyone knows stemming is lossy. We looked at it anyway, because we had a number: across 45,366 inflected English words, the lemma costs fewer tokens than the inflected form about 39% of the time, and about four tenths of a token on average. Then we ran it against twelve real prompts. On six of seven tokenizers the saving was zero. Not small. Zero.

Both numbers are correct. They answer different questions, and the gap between them is the useful part of this post.

The sweep said yes

We already had the measurement rig. Building the rule audit meant measuring 150,673 dictionary entries in their in-context form against seven tokenizers — o200k, cl100k and r50k from tiktoken, Llama 3.2, Qwen 2.5, DeepSeek-V3, and Claude via count_tokens. Asking whether a lemma is cheaper than its inflected form is one query against that table.

45,366 of those entries — 30.1% — are inflected forms. For each one we compared the cost of the word against the cost of its lemma.

That is a promising table. Four tenths of a token per swap, and it almost never backfires. We wrote the tier.

Twelve real prompts, zero tokens saved

Then we ran it on the corpus our other compression harnesses use: twelve prompts shaped like the traffic we actually serve — system prompts, questions about an API, meeting notes, a URL or two. Naive lemmatization fires 64 times across them, 45 distinct word-to-lemma pairs. We tokenized the original and the rewritten text in full, with the real tokenizers, and counted.

The zero is not three savings cancelling three losses. Every one of the 64 rewrites is individually one token in, one token out. We checked it the hard way as well as the easy way: aligning token offsets against the rewritten span inside the full prompt, so a word is measured where it actually sits rather than in isolation. All 64, including https → http inside a URL and the one sentence-initial capital. One to one, every time.

It is also robust to the choice of lemmatizer. Verb-first, noun-first, shortest-lemma-of-any-part-of-speech: all of them produce zero on all six.

A dictionary gives every word one vote

The sweep and the corpus disagree because the sweep counts word types and a prompt is made of word occurrences. In a dictionary, aardwolves and years get one vote each. In English, one of them turns up rather more often than the other.

Weight the same 45,366 pairs by how often the words actually occur — we used a frequency table built from 98.5 billion occurrences — and the mean saving collapses by roughly a factor of twelve, from +0.40 to +0.033 tokens per swap. The share of real swap occurrences that save anything falls from 39% to between 4.1% and 5.0%.

Morphological compression works on rare, long, heavily-affixed words. Prompts are not made of those. They are made of the few thousand forms that appear constantly, and those are exactly the forms a 100,000-to-200,000-entry vocabulary has already spent an entry on. There is nothing left on the table to pick up.

We should add that the mean was the wrong statistic to lead with in the first place. With 59% of swaps saving nothing, the median saving is zero, and reporting +0.40 without the distribution is the same aggregation mistake the post is about. We made it, in a spreadsheet, before anyone ran a prompt.

Claude is the exception, and it charges grammar for it

Claude is the one tokenizer that moves, and it moves for a reason we have written about before: it spends more tokens on ordinary English than the open BPEs do. On these twelve prompts Claude reads 758 tokens where o200k reads 483 — about 1.6 times as many for identical text. More words arriving as several tokens means more words where dropping a suffix crosses a token boundary.

3.03% on the family that is 33% of the prompt tokens we send is not nothing. Here is what it buys.

That is the trade in one line. Of the 64 rewrites, 18 are the copula being flattened — is to be thirteen times, are to be three, was to be twice. All eighteen are Claude-neutral. They cost the tense of the sentence and save nothing at all. Forty of the rewrites change the token count by exactly zero, and one of them, following → follow, makes the prompt more expensive.

So the honest version of the Claude result is that a 3% saving is available if you are willing to send a model text with the verbs broken, and roughly two thirds of the damage buys nothing even in principle.

There is no single naive lemmatizer

This one caught us during review, and it is worth stating because it is the kind of error that survives into a published chart.

A lemmatizer needs to know the part of speech. Without a tagger you have to pick a default, and the two obvious defaults fail differently.

The noun-only branch produces strings that are not words. wa costs Claude two tokens where was costs one, so the rewrite is a straight loss in both meaning and money. But you cannot cite was → wa and following → follow as failures of the same system, because no single configuration produces both. Our first draft of this post did exactly that. A part-of-speech tagger would fix the ambiguity and costs more than the 3% is worth.

One example survives either way: pass → pas. The verb lemma leaves it alone, so the noun branch gets it regardless. Two tokens before, two tokens after, and a word that no longer exists.

What stemming actually destroys

Stemming is cruder than lemmatization by design: it chops affixes without checking whether the result is a word. Over our lexicon, 41.7% of entries — 62,847 of 150,673 — stem to a string that is not itself in the dictionary. ably becomes abli.

We expected those non-words to tokenize badly, and measured it to make the point. The measurement said otherwise, so here is the real result: non-word stems still save tokens on average, about 0.18 of one. They just save far less than stems that are real words, which average 0.53 — and they backfire at 12.5% against 0.9%, roughly fourteen times the rate. Stemming is not bad at saving tokens. It is bad at everything else.

What it is unambiguously bad at is remaining reversible. 31,315 stem groups in our lexicon contain more than one distinct word. abolish, abolishable, abolished, abolisher, abolishes, abolishing, abolishment and abolishments all become one string. Nothing downstream can tell them apart again, because the information is gone rather than compressed.

Andes, Wales, Philippines

Proper nouns are where this stops being a trade-off and starts being a bug. We assembled 48 entities — surnames, companies, products, places, a few of our own systems — and ran both algorithms over them. 41 of the 51 whitespace-separated tokens were altered.

That set is ours and it is not a random sample — we deliberately chose names that look inflected, because those are the ones that break, and 80% is a property of the list rather than of English. The point survives the disclosure. A compression stage that rewrites a customer's name, a product, or a region is not making a trade you get to tune. Our current engine has no word-level morphological rewriting at all, and the span-preserving guard that protects code, URLs and numbers has no clause for proper nouns, so this would have had to be built before the tier could ship safely. It would have cost more than the tier was worth.

Search can afford this. Prompts cannot.

None of the above is a criticism of stemming. It is the right tool for the job it was built for, and that job has a property prompt compression does not: the output is an index, not a message. If abolishes and abolishment collapse to the same key, retrieval improves, and nobody ever reads the key. The lossiness is the feature.

Send the same transform to a model and the collapsed form is the message. The reader is a system that infers from tense, number and specificity — and we already know from our own evaluation loop what that costs. Compression currently makes about one answer in seven measurably worse. A tier that damages grammar for a 3% saving on one tokenizer family, and 0% on the rest, cannot clear that bar. We did not ship it.

What we changed

Not the engine. The evaluation.

A dictionary sweep is a hypothesis generator, never a green light. It answers "could this ever help", which is a different question from "does this help on my traffic", and the two can differ by a factor of twelve. Nothing gets built off a type-weighted table again without a frequency-weighted or corpus-level check first.

Report the distribution, not the mean. A mean of +0.40 with a median of zero is a sentence about a long tail wearing the costume of a typical case.

Measure the whole text, not the parts. Summing per-word measurements is a proxy. It happened to be additive here, and we only know that because we checked it against the full prompt.

The negative results are cheaper than the positive ones and we would rather publish them than repeat them. This one cost a few hours of measurement against a rig that already existed, and it closed a direction that looked obviously correct on paper.

Measured against engine 3.2.24 and the TokenLens corpus in August 2026. Claude figures use count_tokens on claude-opus-5 with the message envelope subtracted; Haiku uses an older tokenizer and is not covered. The twelve-prompt corpus is small, and "zero" is a fact about those 64 rewrites rather than a law of English — but it is zero on every tokenizer we could run locally, under four different lemmatizer configurations, measured on the full text.

Try Nyquest →

No account needed to start chatting — or bring your own API key.

← All posts