Characters Are Not Tokens

Every prompt-shortening guide eventually suggests the same thing: replace long words with short symbols. Write & instead of and. Write w/ instead of with. It looks like free money — the text is visibly shorter, and you are billed by the token, so shorter must be cheaper.

We measured it. Here is the whole result:

"and" → "&", in context

cl100k (GPT-4 class): 3 tokens → 3 tokens
o200k (GPT-4o class): 3 tokens → 3 tokens
Claude: 3 tokens → 3 tokens

Two characters removed. Zero tokens saved. On every tokenizer, bare and in context.

Why it was never going to work

BPE builds its vocabulary by repeatedly merging the most frequent pairs it sees. and is one of the most common words in English, so it becomes a single token almost immediately — it is not assembled from pieces at inference time, it is simply one entry. & is also a single token. Swapping one for the other trades a token for a token.

The same is true of nearly every single-word substitution people reach for. We tested thirty-two of them. Fifteen saved nothing at all: or to |, not to !, at to @, percent to %, plus to +, maximum to max. All zero.

Nine of them made it worse

This is the part that should give you pause, because these are the substitutions that look most obviously correct:

with → w/ — costs one token
without → w/o — costs one token
for example → e.g. — costs one token
that is → i.e. — costs one token

The slash and the full stops force pre-tokenizer splits. w/o is three characters and three tokens; without is seven characters and one. You shortened the string and lengthened the bill.

What does work

The savings track words removed, not characters removed. A phrase collapsing to a symbol wins roughly one token per word it drops:

greater than or equal to → >= — saves 4
is equal to → = — saves 2
not equal to → != — saves 2
greater than → > — saves 1

Mathematical glyphs behave better than folklore suggests, incidentally. ≥ and ≠ are one to two tokens and save real money against the phrases they replace. ∴ (therefore) and ∵ (because) cost one token more than the plain English word. Measure the specific glyph; do not assume Unicode is expensive or cheap as a class.

The prize is structural, not lexical

All of the above is rearranging deck furniture. Here is where the actual money is:

The same three records, two encodings

As JSON: 25 tokens
As YAML: 3 tokens

An 88% reduction, on identical information. Punctuation is what costs under BPE — every quote, colon, comma and brace tends to be its own token, and quoted keys split further. JSON is a format optimised for unambiguous parsing, and it pays for that in tokens. Converting a table of records to YAML or CSV before it reaches the model is worth more than every word-level rewrite in this post combined.

On a real fifteen-row inventory payload through our engine, that conversion took the request from 345 tokens to 159.

The bug we found on the way

In the spirit of honest numbers: our own JSON converter had not run for three weeks.

We protect code blocks from compression by masking every fenced block before the rules execute — indentation is semantics, and a rewrite that reflows YAML changes what it means. The JSON converter matches ```json fences. It ran after the mask. So it was searching for exactly the thing that had been replaced by a placeholder fifty-five lines earlier, and it had been quietly returning its input unchanged since the day the protection shipped.

Two safety features, each correct alone, silently cancelling. The converter now runs before the mask and re-fences its output so the protection still applies.

Why nobody noticed

Here is the thread connecting all of it. If you measure prompt size in characters — or in any character-derived approximation — then every one of these transformations looks like a win. and to & removes two characters. with to w/ removes three. A dead converter that returns its input unchanged removes zero, which at least reads as neutral.

A character-based metric reports a saving for any rewrite that shortens text. There is no input for which it says "that made things worse." It is not merely imprecise; it is structurally incapable of failing, which means no test built on it can catch a regression.

That is the actual lesson, and it generalises well beyond tokenizers: a metric that cannot report a loss is not measuring the thing you care about. If your optimisation dashboard has never shown a negative number, that is not evidence the optimisation works.

The numbers behind this post

32 word-and-phrase substitutions measured against three tokenizer families, in isolation and in sentence context
Result: 8 save tokens, 15 save nothing, 9 cost tokens
Structural conversion measured at 25 tokens → 3 on identical data; 345 → 159 on a real payload
7 new regression tests, including one asserting that single-word symbol swaps cannot be reintroduced, and one asserting prose such as "less than ideal" is never mangled into an operator

If you want to check any of this, you do not need us: tiktoken is a public library and the encodings are published. That is rather the point. The tokenizer is not a mystery — it is just something almost nobody actually measures before optimising against it.

Try Nyquest →

No account needed to start chatting — or bring your own API key.

← All posts