How it works · one pipeline · every request
Every request flows through the same pipeline: authenticate, classify, optionally enrich with memory and web context, compress, dispatch to the right model, stream the response back. Visible. Auditable. Overrideable at every step.
Step 01–06 · request lifecycle
Six stages on the request path. Each one is observable in the inspect panel.
JWT for the web app, nq-v1-... for API integrations, or BYOK
keys for direct provider access. Each credential type has different
statefulness — JWT requests carry memory, API-key requests are stateless by
design.
The AutoRouter classifies the request — complexity and domain (code, reasoning, creative, vision, general) — and picks a tier (T0–T3) and model. Model choice inside the tier is informed by judged-quality scorecards kept per model, per domain, refreshed continuously from graded real answers. Circuit breakers track provider health and fail over automatically.
For app sessions, the memory system embeds your message and vector-searches your fact store for relevant prior context, scored by confidence. Sanitized facts are injected into system context. API-key requests skip this step.
If the message benefits from current information, a detector triggers web search. Top results are fetched, content-type-gated and size-capped, then stitched into the prompt. Sources are recorded for provenance.
The compression sidecar applies regex rules, prunes turn history above the context cap, and optionally invokes a semantic LLM stage on long system prompts. Token telemetry is captured. Tool schemas pass through untouched.
The optimized request goes upstream through the provider layer — OpenRouter, NVIDIA Build, or Nyquest's local tier — to the selected model. Tokens stream back via SSE, with response headers carrying telemetry: original tokens, compressed tokens, savings %, web-grounding sources, and the model that actually served the request.
Routing layer
The dispatcher is what makes Nyquest a routing platform vs. a direct proxy. Every chat request is classified to a tier, with explicit defaults per plan.
| Tier | Default Model | Available On | Use Case |
|---|---|---|---|
| T0 | local (llama 3.2) |
All accounts | The simplest requests are served on Nyquest's own hardware at $0 when the local tier is available — streaming included. |
| T1 | claude-haiku-4-5 |
Pro | Default cheap path. Fast iteration, simple questions, quick drafts. |
| T2 | claude-sonnet-4-6 |
Pro | Pro default for complex tasks. Strong reasoning, instruction-following. |
| T3 | claude-opus-4-6 |
Pro | Heaviest workloads. Reasoning-intensive analysis, long-horizon planning. |
| Free chain | rotating free models |
All accounts (free mode + failover) | A provider-diverse chain of currently-free models, live-probed around the clock and rebuilt automatically when the free ecosystem shifts. |
Free accounts route to whichever models are currently no-cost in the catalog (the available set rotates), plus BYOK with their own provider keys. The dispatcher tiers above apply once an account flips to Pro on first wallet funding.
If a tier fails N times in a row (default 5, configurable), the dispatcher opens the breaker for 60 seconds and routes to the next tier. Health is tracked per-tier and visible in admin.
Pin any specific model from the picker and bypass the dispatcher entirely. The pinned model handles the whole conversation — no automatic switching.
Every request shows which tier was selected, which model actually served it, why, and how many tokens it cost. Visible in the inspect panel and usage analytics.
Self-learning routing
Routing isn't a static table. A sample of real answers is independently graded by an AI judge on factuality, consistency, completeness, and hedging — and those grades flow straight back into which model gets your next request.
Quality is tracked per model, per domain — the model that writes the best code isn't the one that reasons best. Scores are recency-weighted with a two-week half-life, so silent provider upgrades (and regressions) show up in routing within days.
When a proven winner exists, you get the proven winner. A small, strictly capped exploration budget goes to the most promising under-tested candidates, allocated by upper-confidence-bound (UCB) bandit math — the same class of algorithm behind large recommender systems.
Models that upstream providers list but can't actually serve are detected on first failure and removed automatically — and re-enabled automatically when they come back. The free-model chain is live-probed every few hours. What's in the picker works.
Memory subsystem
Conversation history is the literal messages. Memory is a curated, embedding-indexed store of facts extracted across conversations.
After a conversation turn, an LLM-driven extractor distills noteworthy facts. Output is structured: subject, predicate, confidence, source.
Before storage, facts are scrubbed for PII, medication names, and numerics that look like keys or credentials. Anything ambiguous gets dropped, not stored.
Each fact carries a 0..1 score. Newer and more-corroborated facts rank higher. Stale or contradicted facts decay.
At chat time, your message is embedded and the top-K facts are retrieved by vector search. Only the most relevant ones are injected into context.
Every recalled fact records what triggered its retrieval and which conversation it came from — auditable in the grounding-events log.
Memory recall fires only on JWT (web app) requests. API-key requests skip recall entirely — clean developer-API behavior.
Web grounding
When a chat needs current information, the web subsystem fetches actual page content — not cached snippets, not summaries from a middleman.
No third-party search API key required. Toggle web grounding per-request via header, or let the detector decide.
Under the hood · step 05 in depth
Stage 05 of the lifecycle is a separate Rust service that runs as a sidecar to the backend. Its v3.2.8 release is open source and self-hostable; the hosted platform runs a later, proprietary build.
Whitespace cleanup, markdown noise removal, tokenizer-unfriendly character handling. Reduces entropy before any other stage runs.
532 regex rules across 19 categories: filler removal, verbose-phrase
substitution, redundancy collapse, structural noise. A RegexSet
prefilter and zero-allocation Cow<str> rule misses keep the
hot path sub-millisecond on typical prompts.
Trims older turns based on a token cap (50K on prod) while preserving the last 4 turns verbatim and a 6-turn minimum. Critical instructions stay in scope.
For long history or large system prompts, an LLM rewrites and condenses while preserving meaning. Falls back to extractive (rule-based) on timeout.
Per-LLM hooks adapt compression to the target provider's tokenizer and prompt conventions. Tool schemas, image blocks, and audio blocks pass through untouched.
Every call returns original tokens, compressed tokens, savings %, latency, and which stages fired. Visible to admin and inspect-panel users.
Open-source at Nyquest-ai/nyquest-rust-fullstack-3.2.0 · Docker image at ghcr.io/nyquest-ai/nyquest-engine · MIT · v3.2.8
Performance
Measured on the v3.2 Docker container with --network host.
Sub-millisecond hot path; cached identical content returns in microseconds;
semantic stage adds latency only on long inputs.
| Test | Result | Detail |
|---|---|---|
| Health throughput (single thread) | 8,266 req/s | p50 <1ms · host networking |
| Concurrent (20 workers, 5000 req) | 18,287 req/s | p50 1ms · p95 1ms · p99 2ms |
| Cache hit (warm replay) | ~3,730× | ~70.9 ms cold → ~19 µs hit |
| Failed requests | 0 / 10,000 | across single + concurrent runs |
| Memory footprint | 48.5 MB RSS | 15 threads · Docker container |
| Production, live traffic (measured July 2026) | 33% avg token savings | 30-day window · grows 15% → 75% as conversations run 7 → 19 messages |
Reliability
Compression isn't just about cost. Cleaner inputs reduce the conditions that cause models to hallucinate.
The concept
Same principle as audio compression. Preserve the frequencies that carry meaning. Remove the noise that doesn't.
Open the app and send a message. The full pipeline runs on every request, and the inspect panel shows you what happened.