A MAC Address Costs More Than This Sentence

Here are two strings, measured on the tokenizer behind GPT-4-class models:

A third of the bytes, twice the tokens. The English sentence runs at 6.25 bytes per token; the MAC address manages 1.06, which is another way of saying the tokenizer has essentially given up and is spelling it out one character at a time.

Here is what that looks like from the inside:

An IPv6 address, a UUID and a switch interface. Every one of them is a single identifier to the person reading it and to the system that emitted it. None of them survives contact with the tokenizer as fewer than ten pieces.

The obvious fix does not work

The intuitive response is that the vocabulary is too small. Give the tokenizer more room and it will learn these patterns.

So we measured the same strings against a vocabulary twice the size — 100,277 entries versus 200,019.

Ninety-nine thousand additional vocabulary slots. Not one of them went to the MAC address, the IPv6 address, the UUID, the interface name, the IPv4 address or the timestamp. The URL got longer.

That is the result that made us stop and look at the algorithm, because it rules out the easy explanation. This is not a capacity problem. It is a selection problem.

What BPE is actually choosing

Byte-pair encoding builds its vocabulary with a loop short enough to write out in full. Start with the 256 possible bytes. Count every adjacent pair in the corpus. Merge the most frequent one. Add it to the vocabulary. Repeat until the vocabulary is the size you asked for.

Read that again and notice what the selection criterion is. It is frequency. That is the entire decision. Whichever pair occurs most often wins the next slot, and nothing else about the pair is consulted.

Which explains the numbers above rather neatly. 3C and 4D are not frequent in general text. Neither is f9a0. They lose every round to the pairs inside ordinary English words, forever, no matter how many rounds you run — and adding another hundred thousand rounds just hands more slots to the things that were already winning. Meanwhile the structure that a human would call obvious — that a MAC address is six hex pairs separated by colons, that it is one thing rather than sixteen — is information the algorithm never had a way to use, because it only ever asked one question.

Frequency is one signal

There are others, and they are not exotic:

The distinction between frequency and mutual information is the sharp one. A pair can be frequent because both halves are independently common, which tells you nothing — or frequent because the halves genuinely belong together, which is exactly what you want a vocabulary entry to capture. Raw frequency cannot tell those two cases apart. PMI is the standard way to ask.

So the hypothesis is narrow and testable: merge selection informed by more than frequency produces a better tokenizer. We are calling the thing that tests it NyBPE, and it runs on a lab box we have named ember.

Fewer tokens is not the win

This is the part we want on the record before any results exist, because it is the trap.

Token count is trivially gameable. Make the vocabulary bigger and the count goes down. Merge more aggressively and it goes down further. A tokenizer that shreds nothing and swallows everything would post the best compression number in the benchmark and could easily be worse for the model — because the point of a token is not to be large, it is to be a unit the model can learn something about.

Readers of this blog may recognise the shape of that argument. We wrote in Characters Are Not Tokens that a metric which cannot report a loss is not measuring the thing you care about. Token count on its own is that metric again, wearing a different hat. If the only number we publish is "NyBPE emits fewer tokens", we have learned nothing and you should not believe us.

So the benchmark was built first, before a single line of the new algorithm, and a tokenizer only counts as better if it improves something without wrecking something else:

The measurement that decides it

The one that matters most is the least convenient. You cannot compare perplexity between two models that use different tokenizers — the number is per-token, and if the tokens differ the two figures are not the same quantity. Compare them anyway and the tokenizer producing fewer, fatter tokens tends to look better while telling you nothing.

The honest comparison normalises to the raw text: bits per byte. Same corpus, same architecture, same parameter count, same optimiser, same steps, same seed, and the question is how many bits each model needs to express the same underlying bytes. That is a number a tokenizer cannot flatter itself into.

That means NyBPE is not finished when it compresses well. It is finished when a small model trained on it needs fewer bits per byte than the same model trained on standard BPE — or when it does not, and we say so.

Experiment 001

The first experiment does not try to build the ambitious version. It asks whether the premise holds at all, by changing exactly one thing — the scoring function — and leaving everything else identical:

The control matters more than it sounds. We are reimplementing ordinary frequency BPE ourselves rather than importing one, because otherwise every difference we measure could be an artefact of two different codebases rather than the merge rule we changed. The control has to be the same program with one function swapped.

And Experiment 001 is allowed to fail. If information-aware scoring turns out to be a rounding error against plain frequency, that is a real answer arriving cheaply, before any of the more interesting architecture gets built on top of an assumption that was never true.

What we have not done

In the spirit of the numbers in this post: everything above the Experiment 001 section is measurement, and everything in it is a plan.

NyBPE has produced no results. There is no tokenizer to download, no benchmark table with our name winning it, and no claim here that we can beat byte-pair encoding — only that the frequency rule looks like it leaves something on the table, and that the gap between an English sentence and a MAC address is large enough to be worth a lab.

The entropy-aware, domain-aware and adaptive work described in our internal plan stays on the shelf until Experiment 001 produces data. Building all of it first and then measuring would guarantee we found something to be pleased about.

You can verify the measurements in this post yourself in about five minutes. tiktoken is a public library, the encodings are published, and the strings are all up there. That is rather the point — the tokenizer is not a mystery. It is just something almost nobody looks at before building on top of it.

Try Nyquest →

No account needed to start chatting — or bring your own API key.

← All posts