What Is LLM Routing? Cut AI Costs Without Losing Quality
LLM routing is the practice of sending each request to a model chosen for that task, rather than sending everything to the same model. Done well, it cuts cost per task while keeping quality where it matters, because most steps in a product do not need a frontier model.
Key takeaways
- LLM routing chooses a model per request or per step instead of using one model for everything.
- Cascade drafts with an efficient model and escalates only when a quality check rejects the draft.
- Critique uses one model to draft and another to review, then revises the output.
- GitHub reports 47 models used in September and no single model above 25% of requests, so multi-model use is normal.
- Measure cost per successful task and quality on your own eval set, not on a leaderboard alone.
LLM routing is the practice of sending each request to a model chosen for that task, rather than sending everything to the same model. The point is not to find the cheapest model. It is to spend on the steps where quality changes the outcome and save on the steps where it does not. A support classifier and a customer-facing answer have different tolerances for error.
TL;DR
- Routing picks a model per request or per step instead of using one model for everything.
- Cascade uses a cheap model first and escalates only when a check fails.
- Critique uses one model to draft and another to review, then revises.
- Multi-model use is already normal: GitHub reports 47 models used in September and no model above 25% of requests.
- Test on your own tasks and track cost per successful task, not cost per token.
Disclosure: ZeroRaised publishes this guide and is a launch site for bootstrapped products. Facts come from the linked sources as checked on 30 Oct 2026; ZeroRaised has no commercial relationship with the companies named.
LLM routing is a control layer that decides which model handles a request, based on the task, the difficulty, the cost budget or the result of an earlier attempt.
Why does LLM routing matter now?#
Model menus got large. GitHub's Octoverse 2026 report says 47 distinct models were used in September, that 61% of users who picked a model used more than one, and that no single model had more than 25% of requests. The request mix was 38% general purpose, 37% deep reasoning and 25% fast models, and 63% of paying users used automatic model selection (GitHub Octoverse 2026).
Most people do not want to choose a model for every call. They want the system to choose, and the bill to reflect it. Our Octoverse breakdown covers what the same report says about agent-started pull requests.
GitHub ships that idea as a product. Copilot Fusion is generally available and routes and orchestrates models per task, with Standard, High and Max effort levels (GitHub Copilot Fusion changelog). Its results are instructive: Fusion Max scored 60 on the Artificial Analysis Coding Agent Index, within two points of Claude Code with Opus 5.5 Max, at about half the API cost per task, about half the tokens and about 40% less time (GitHub Copilot Fusion changelog).
Cascade or critique: which pattern fits?#
Fusion names two patterns worth stealing.
Cascade starts with an efficient model. A quality check then accepts the draft or escalates it to a stronger model (GitHub Copilot Fusion changelog). Cascade is the cost pattern. It assumes most requests are easy and a minority need the expensive model.
Critique uses one model to draft, another to review, and then revises the draft (GitHub Copilot Fusion changelog). Critique is the quality pattern. It assumes errors are expensive enough to justify a second pass, and that a fresh model sees mistakes the first one missed.
The trap is using critique everywhere. Two calls per request doubles token spend before you improve anything. Use critique where a wrong answer costs money or trust. Use cascade where the work is repetitive and a failed draft is cheap to detect.
How do you route requests in your own app?#
You do not need a router vendor. The pattern fits in ordinary code.
- Label the task. Give each call a task type: classify, extract, draft, reason, summarise. The label is the routing input.
- Set a quality floor per task. A classifier might need 95% agreement on your sample; a marketing draft a human read. Write the threshold down.
- Pick a default cheap model per task. Start low and escalate on evidence.
- Build a check that can fail. For extraction, validate against a schema. For code, run tests. For prose, use a rubric or a second model. A check that always passes is not a check.
- Escalate only on failure. Send the failed case to the stronger model and log why.
- Track cost per successful task. This is the only number that tells you routing is working. Token price alone misleads.
- Keep a fallback. If the cheap model is down or the check errors, fail cleanly or fall back to a known model.
The LLM cost guide covers caching and prompt trimming, which compound with routing. The LLM API comparison covers shortlisting models.
What does routing look like with a real model menu?#
Mistral Large 4 shows a single model trying to cover several tiers. It has three reasoning modes, Low, Medium and High, with High as the default (Mistral). On the API it costs $0.70 per million input tokens, $0.07 per million cached input tokens and $2.10 per million output tokens (Mistral).
Mistral calls it the best open-weight model for coding, citing 19.7% on FrontierCode 2.0, a private benchmark from Cognition (Mistral). On the Artificial Analysis Coding Agent Index it scored 48%, effectively tied with GLM-5.3 while using 47% fewer tokens per task (Mistral). Fewer tokens per task is a routing advantage: the same job costs less at the same per-token price.
The licence matters if you host rather than call the API. The weights are under the Mistral Ecosystem Open-Weight License, a modified MIT licence that allows use, modification, fine-tuning and commercialisation, provided that running a Model-as-a-Service business needs a separate commercial licence with Mistral (Mistral). The open-weight versus closed model guide covers the trade-offs.
How do you know routing is not hurting quality?#
Benchmarks are a starting point, not an answer. Fusion Max landing within two points of a frontier setup is encouraging, but your product is not that index (GitHub Copilot Fusion changelog). A private benchmark such as FrontierCode 2.0 is also hard to reproduce (Mistral).
Build a small eval set from real requests. Fifty to a hundred cases with known good outputs will tell you more than a leaderboard. Run the routed pipeline and a single strong model on the same cases. Compare pass rate, cost per successful task and latency. If the routed version is within your quality floor and materially cheaper, keep it. If not, the router is not ready. Our guide to evaluating LLM outputs covers the mechanics.
Watch two failure modes. Escalation creep, where the check starts rejecting more drafts over time, quietly turns a cascade into two calls per request. And silent quality drift, where a provider updates a model and your check no longer reflects the output. Pin versions, and re-run the eval when anything changes.
Where ZeroRaised fits#
ZeroRaised is a launch and discovery site for products built without institutional venture capital or accelerator equity. If you have a product people can try, submit your product; the pricing page explains the current options. For more analysis like this, join the ZeroRaised newsletter. No promises of traffic or results, just the useful parts.
Sources#
Frequently asked questions
What is LLM routing in simple terms?
LLM routing is choosing which model handles a given request or step, instead of sending every request to one model. A router might pick a fast, cheap model for classification and a stronger model for a hard reasoning step, or escalate automatically when a draft fails a check.
What is the difference between cascade and critique routing?
In a cascade, an efficient model drafts and a quality check either accepts the draft or escalates to a stronger model. In a critique workflow, one model drafts, another reviews it, and the draft is revised. Cascade is usually about cost; critique is usually about catching errors.
Does routing always reduce cost?
No. A badly tuned router adds overhead, retries and duplicated work. Cost falls when most traffic goes to a cheaper model and escalation is rare. If your quality check rejects too many drafts, you pay for both the draft and the stronger model.
Do I need a separate router service?
No. You can route with a few lines of application code: classify the request, pick a model, call it, and escalate on failure. A hosted router is convenient, but the important part is the decision rule and the quality check, not the vendor.
How do I test whether routing hurt quality?
Keep a small set of representative tasks with known good outputs. Run the routed pipeline and the single-model baseline on the same set, compare pass rates and cost per successful task, and re-run when models or prompts change.