Every prompt that flows through Nyquest passes through our compression engine first. It strips filler, collapses redundancy, and trims the tokens we pay providers for — it's the quiet layer that makes our economics improve with scale. But until this week it optimized exactly one number: tokens removed. It had no idea whether a compression ever hurt an answer.
That's a dangerous thing to be blind to. So we built the feedback loop — and within a day, it caught a real failure mode, we shipped the fix, and we caught a bug in our own measurement for good measure. Here are the honest notes.
A self-improving compressor isn't one that compresses harder. Chase the savings number alone and "self-improving" silently becomes "self-degrading" — the dashboard looks great while answers quietly get worse. The real objective is: the maximum compression that keeps the answer as good as the uncompressed one would have been, learned per kind of content. Everything below serves that.
For a sampled fraction of already-compressed chats, a background task re-runs the original, uncompressed prompt against the same model, and an independent LLM judge compares the two answers: equivalent, compressed_worse, or compressed_better, with a 0–100 score. Three properties we consider non-negotiable:
Rather than wait for organic traffic, we ran a controlled battery: 12 prompt archetypes — verbose filler, facts buried in padding, code, exact dollar figures, strict format constraints, creative tone, multi-step math, and a short control — across two models on two different routing paths. Every run shadow-judged. The result was a clean, two-sided verdict on our engine:
The insight that matters: it's not how much you compress — it's what gets touched. Prose is robust even under heavy compression. Code and numbers are fragile under any compression at all. A rule pipeline built for prose was, occasionally and silently, collapsing the indentation of a Python snippet (fatal — indentation is semantics) or nudging a figure in a revenue table.
Engine 3.2.9 ships verbatim-preserve spans. Before the rule pipeline runs, we mask everything fragile — fenced code blocks, indented code lines, inline code, currency amounts, and any digit-bearing token (IDs, dates, times, figures) — behind sentinels drawn from Unicode's private-use area, characters no text rule can ever match. The rules do their work on the prose around them; then the spans are restored byte-identical. And it fails closed: if a sentinel ever goes missing, the engine returns the original text uncompressed rather than guess.
Post-fix verification, same battery payloads: code blocks byte-identical (fenced and unfenced), every dollar figure and ID verbatim, prose still compressing. One deliberate trade came with it: we no longer rewrite dates into ISO form — preserving January 14, 1999 exactly beats saving two tokens on it.
Honest engineering notes include this section. After the fix, the judge still flagged code prompts as worse — even though we could prove the compressed prompt was byte-identical where it mattered. The culprit was the eval itself: our shadow baseline didn't replicate the original request's max_tokens. The user's capped, truncated answer was being judged against an unbounded full-length answer — and losing on completeness, not compression. One plumbing fix later, the same archetypes judged equivalent-or-better at 100, and repeat runs scattered exactly like sampling noise should.
Lesson worth sharing: when you build an eval, budget time to evaluate the eval. Ours was three days old and already had opinions that weren't true.
The shadow eval now samples production traffic continuously, building a per-segment safety map: which content classes compress safely, at what levels, on which models. That map is what lets the next phases be data-driven instead of guesswork — per-segment compression levels, per-rule attribution and demotion, and eventually a small compressor model distilled from our own (original, safe-compression, outcome) triples. A compressor trained on your own traffic is a moat a competitor can't copy without your traffic.
Try Nyquest → No account needed to start — or bring your own API key.