Our latest GPT-6 Astra test cut input from 4,210 to 181 tokens: a 95.7% reduction, with the correct answer preserved. This was a retrieval task, meaning a question answered from a collection of documents or records. Nyquest selected the relevant source material instead of sending the full collection to the model.
Claude Fable 5.1 input fell by 92.1% on its retrieval task, 44.5% across a three-turn history session, and 20.6% across a three-turn tool JSON session. Gemini 3.1 Pro Preview retrieval input fell by 96.3%.
The useful change is what sits behind those numbers. The request compiler now makes better decisions about cache eligibility, estimates the complete request for each supported model and provider, and follows responses through schema checks and billing reconciliation. We tested the conversation after compression, including the turn that asks for a source we previously left out.
This is a small internal MVP evaluation using synthetic data and short factual answers. Below are the improvements, the measured gains, the unchanged controls, and the work still ahead.
1. Cache eligibility changed the decision
A smaller request can cost more when it replaces a warm cached prefix. Our cost guard accounts for that. Previously, however, it could compare a best-case cached original with an uncached compressed request even when the exact pinned route had never opted into caching. That assumption prevented useful Fable compression.
For supported, explicitly pinned Anthropic requests without cache opt-in, the compiler can now evaluate an uncached scenario. Unknown routes, implicit caches, presets, and explicit cache markers keep the conservative treatment. Explicit cache configurations also account for the higher cost of cache writes. OpenRouter documents the provider-specific cache eligibility and write rules.
We also found a mismatch further down the pipeline: the gateway could add a cache marker after the compiler made its decision. Eligible typed uncached requests now remain uncached through dispatch. Caller-requested caching still works, and the guard remains in place.
To isolate the software improvement, we ran the previous deployed binary and the new compiler against the same new Fable retrieval, history, and JSON fixtures. The old version kept all three originals. The new version selected all three compressed candidates. This comparison required no additional model calls.
The transformations are structural. Retrieval and history select whole relevant sources and preserve recovery paths. The tool JSON path removes whitespace while preserving values, number spellings, key order, and the tool schema. It does not depend on a paid model call to prepare the smaller request.
2. The measured gains, including recovery
Input counts below come from provider-native usage. Each row compares a baseline with its candidate on the same model and workload. History and JSON rows are cumulative three-turn sessions; retrieval rows are single request pairs.
| Model / workload | Input tokens before → after | Input reduction | Provider cost reduction | Net wallet charge reduction |
|---|---|---|---|---|
| Claude Fable 5.1 Retrieval | 5,573 → 440 | 92.1% | 90.2% | 78.2% |
| Claude Fable 5.1 3-turn history | 7,788 → 4,325 | 44.5% | 42.6% | 42.3% |
| Claude Fable 5.1 3-turn tool JSON | 10,623 → 8,436 | 20.6% | 20.0% | 20.0% |
| GPT-6 Astra Retrieval | 4,210 → 181 | 95.7% | 94.8% | 37.7% |
| Gemini 3.1 Pro Preview Retrieval | 4,102 → 152 | 96.3% | 68.5% | 20.5% |
Those three percentages measure different things. Input reduction measures the prompt tokens removed. Provider cost is the observed upstream bill, including output, reasoning where billed, and cache effects. Net wallet charge is what the internal test account paid after managed-chat billing and capped savings credits. Our pricing page explains the charging policy; none of these columns should substitute for another.
The history test is especially useful. It included an earlier actual answer, a correction that superseded an earlier fact, and a request to recover an omitted source. On the recovery turn the compiler conservatively restored the full history and saved no input. The 44.5% Fable result includes that full-cost turn.
The JSON session used ordinary two-space formatting for 40 inventory items. It tested an initial lookup, an unchanged snapshot repeat, and a later lookup. Each side of a session carried its own actual earlier answers.
We also kept the negative controls. Astra and Gemini history and JSON requests did not clear the cache guard, so their originals were retained. Astra's JSON bill fell 72.0% with zero input tokens removed, mainly because of cache order. Its history bill rose 203.9% with unchanged input. Gemini's cost differences reflected output variation; its history input differed by six tokens because the earlier answers had different formatting. None is a compression win.
An explicit Fable cache control reinforced the point: a cold request cost $0.05584, its warm repeat cost $0.00227, and a changed prefix cost $0.05599. All three retained the original request. Preserving a working cache can be the right optimization.
3. Count the complete request
The provider receives more than the visible prose: role framing, tool definitions, and output schemas also occupy context. We now calibrate complete-request estimates for exact model/provider combinations and retain the raw estimate and calibration revision for inspection.
On fresh requests eligible for this pilot, aggregate count error improved across all three models:
| Model | Eligible calls | Raw estimate error | Calibrated error |
|---|---|---|---|
| Claude Fable 5.1 | 15 | 33.9% | 6.6% |
| GPT-6 Astra | 14 | 31.1% | 15.0% |
| Gemini 3.1 Pro Preview | 13 | 42.7% | 9.7% |
These are labeled estimates. Unsupported routes, opaque inputs, and future model names do not inherit the coefficients. Seven of the 49 calls were outside the pilot, including explicit cache-block requests and short Astra/Gemini candidates. Short requests with substantial schema overhead still need better estimates.
4. Follow each request through the answer and the bill
The live run contained 49 calls: 42 paired requests, three streaming checks, three cache controls, and one diagnostic. All 49 responses passed both the expected-fact checks and strict JSON-schema checks. We reconciled all 49 provider generation receipts and managed settlements through gross debit, savings credit, and net wallet charge.
The new calls cost $0.752768 in actual provider charges. That is the upstream cost of this run, separate from account wallet charges. There were no paid preparation calls or retries.
Failure handling was part of the implementation too. Streaming and non-streaming traces retain the served model, provider, finish state, and refusal state. A refusal does not silently trigger another paid attempt. A reported zero price is valid; an absent price triggers a read-only receipt lookup where available, or explicit reconciliation without an estimated debit. Schema checks do not fetch external references, and unsupported schemas are marked unvalidated. Deterministic tests cover these failure paths; this live run did not need a paid fallback.
Deployment validation also passed 508 Rust tests, 17 database checks, five CI jobs, 33 isolated compiler checks, and nine appliance previews. Those checks support this release's behavior; they do not turn 49 short answers into a broad quality or load study.
For context on the protocol: Fable retrieval used 60 sources; Astra and Gemini used 80 after an initial Astra diagnostic showed that mandatory gateway context erased the smaller fixture's admission margin. We retained that diagnostic in the count and cost. Pair order alternated. A budget stop reduced the remaining Astra session pairs to a 128-token output cap on both sides; Gemini retained 1,024. All final answers completed. Comparisons are within each model and workload, so the table is not a model ranking.
5. What is live, and what comes next
Five premium profiles are enabled in the internal MVP rollout: Fable retrieval, history, and tool JSON; Astra retrieval; and Gemini retrieval. Astra and Gemini history and JSON expansion remains paused while we develop and validate their cache-aware policies. Opus is not included in these validated results.
These changes ship in the request compiler and gateway on the main API and appliance. The compression engine binary remains at version 3.2.27. Improving the decisions around the engine unlocked gains without replacing that binary.
Next we will improve short-request schema and framing estimates, make previews reflect the mandatory context the gateway actually sends, and restore only the history needed to answer a recovery request. Broader quality evaluation and load testing are still required before making customer-scale claims.
The standard for the next iteration stays concrete: a smaller request, the information needed to answer it, and receipts showing what happened. Explore the Nyquest compression engine or read the integration documentation.