What an LLM router saves at November 2026 API prices, the costs list prices leave out, and how to check routing doesn't cut quality before it ships.
An LLM router picks a model for each request: the easy ones go to a cheap model, the hard ones to a strong one. How much it can save depends on two numbers, the share of traffic it moves and the price gap between the two models. At November 2026 list prices, on prompts under 100,000 tokens, Claude Haiku 5.5 costs a twentieth of Claude Sonnet 5.5, so a router that moves 70% of calls from Sonnet to Haiku cuts the bill by 66.5%. A router choosing between Claude Opus 5.5 and Sonnet 5.5, where Sonnet costs half as much as Opus, can never save more than 50%, even if it routes everything.
What routing costs in quality is a separate number, and no router's published results tell you what it is for your application. Those results are averages on public benchmarks: RouteLLM's matrix-factorization router reached 95% of GPT-4's MT Bench score while sending 26% of calls to GPT-4, and an independent 2026 benchmark across 21 datasets found that several routing methods, including the one commercial router it tested, did not reliably beat the best single model. In an agent, routing loses quality on a few routes: a wrong tool call, or an action the model claims and never takes, inside text that reads fine. Averages hide that.
The check is to replay your own production traffic on the strong model and on the cheap one, compare every example per route against limits you wrote before the run, and route only where the cheap model held. The rest of this post covers the kinds of router, what routing saves at today's API prices, the costs list prices leave out, and that check step by step. I build EvalShift, which automates the comparison, but the method works with any tooling.
What an LLM router is #
An LLM router is a layer between your application and the model APIs that chooses which model answers each request. The application sends one request; the router decides whether Haiku, Sonnet or some other model handles it, usually by predicting whether the cheaper model is good enough.
LLM model routing covers several different things, and only some of them route on quality:
| Kind | Decides by | Examples | Routes on quality? |
|---|---|---|---|
| Rule-based or heuristic | Route, metadata, request parameters, a complexity score | Portkey conditional routing, LiteLLM auto routing (default scorer), a switch in your own code |
Only as well as your rules |
| Gateway load balancing | Availability, latency, rate limits, price | LiteLLM's router strategies and fallbacks | No |
| Learned router | A classifier that predicts which model is good enough | RouteLLM, Not Diamond, Requesty, vLLM Semantic Router | Yes, as trained on someone else's data |
| Managed router | The platform's classifier over its own model pool | Azure Foundry model router, Amazon Bedrock Intelligent Prompt Routing, OpenRouter Auto Router | By the platform's quality estimates; OpenRouter ranks by other users' spending |
| Cascade | Try the cheap model first, escalate when a check fails | FrugalGPT | Yes, if the check is reliable |
The difference between an LLM router and an LLM gateway is mostly what the routing decision is
for. A gateway spreads requests across deployments so they stay up and inside rate limits:
LiteLLM's router picks among deployments by simple-shuffle (its default), least busy, lowest
latency or lowest cost, and falls back when one fails. A router in the narrow sense chooses the
model by how hard the request is. Many products do both, including LiteLLM, which added a beta
auto-routing mode that scores each request with a rule-based heuristic by default and sorts it
into simple, medium, complex and reasoning tiers.
A few details from the managed routers matter for testing them:
- +OpenRouter's Auto Router (
openrouter/auto) classifies each request into one of about 30 task types, then ranks models by what OpenRouter's users spent on that task type over the past seven days. It adds no fee, and thedocs ↗ say it can pick a different model on every turn; with a session ID it reuses the earlier model only while that model is still one of the top candidates. - +Azure's model router has three modes. Balanced, the default, picks the cheapest model within 1% to 2% of the top-quality model, Cost widens that band to 5% to 6%, and Quality ignores cost. It routes on the request's text only, and Microsoft updates model version 2025-11-18 in place (docs ↗ ).
- +Amazon Bedrock Intelligent Prompt Routing chooses between two models from one family by predicted response quality, with a "response quality difference" threshold you set. AWS says it is optimized for English prompts only and can't adjust its decisions to application-specific performance data (docs ↗ ).
Two of those three change their behavior without a deploy on your side. OpenRouter's ranking follows a trailing seven days of other people's spending, and Azure's router changes when Microsoft updates it. Keep that in mind for the last step of the check.
LLM API pricing and how much routing saves #
The ceiling on what routing saves is simple arithmetic: saving = share of calls routed × (1 − cheap price ÷ strong price). So the first input is the price gap between the models you would route between.
Prices below are per million tokens, standard tier, read from each provider's pricing page on November 6, 2026 (OpenAI ↗, Anthropic ↗, Google ↗, DeepSeek ↗). The last column assumes 3,000 input and 400 output tokens per call, uncached, roughly one turn of a small tool-calling agent, the same call size as in the cheapest LLM API comparison.
| Provider | Model | Input | Output | 1,000 calls |
|---|---|---|---|---|
| Anthropic | Claude Opus 5.5 | $4.00 | $20.00 | $20.00 |
| Anthropic | Claude Sonnet 5.5 | $2.00 | $10.00 | $10.00 |
| Anthropic | Claude Haiku 5.5 | $0.10 | $0.50 | $0.50 |
| OpenAI | gpt-6.1-terra |
$1.00 | $5.00 | $5.00 |
| OpenAI | gpt-6-luna |
$0.10 | $0.50 | $0.50 |
gemini-3.8-flash |
$0.75 | $3.75 | $3.75 | |
gemini-3.1-flash-lite |
$0.25 | $1.50 | $1.35 | |
| DeepSeek, off-peak | deepseek-v4-pro |
$0.66 | $1.98 | $2.77 |
| DeepSeek, off-peak | deepseek-flash |
$0.15 | $0.60 | $0.69 |
Haiku 5.5's price holds for prompts up to 100,000 tokens. Above that every token in the request bills at $0.50 / $2.50, a quarter of Sonnet's price instead of a twentieth, which matters for an agent deep into a long conversation.
Applied to same-provider pairs, the formula gives the most each router could save at list price:
| Route from → to | Cheap ÷ strong | 50% routed | 70% routed | 90% routed |
|---|---|---|---|---|
| Claude Sonnet 5.5 → Haiku 5.5 | 5% | 47.5% | 66.5% | 85.5% |
gpt-6.1-terra →gpt-6-luna |
10% | 45.0% | 63.0% | 81.0% |
deepseek-v4-pro →deepseek-flash |
24.9% | 37.6% | 52.6% | 67.6% |
gemini-3.8-flash →gemini-3.1-flash-lite |
36% | 32.0% | 44.8% | 57.6% |
| Claude Opus 5.5 → Sonnet 5.5 | 50% | 25.0% | 35.0% | 45.0% |
Three things to read from it:
- +The price gap matters as much as the router. Moving 70% of traffic within the Anthropic pair saves 66.5%; moving the same share between Gemini 3.8 Flash and 3.1 Flash-Lite saves 44.8%. The Gemini row moves on January 1, 2027, when 3.8 Flash's introductory price ends and it goes to $1.50 / $7.50, which takes the 70% saving to 57.4%.
- +Cross-provider pairs need measured cost, not list price. I left them out because the same text is a different number of tokens on each provider. Anthropic says Claude 4.7 and later models count the same text as about 30% more tokens than earlier Claude models, so an LLM cost comparison across tokenizers is off before any routing starts.
- +The 90% column is a ceiling, not a forecast. It assumes every routed call costs the cheap model's list price and nothing else changes. The next section is the list of things that do.
For comparison, here are the published savings claims. LMSYS reported that RouteLLM cut costs by over 85% on MT Bench, 45% on MMLU and 35% on GSM8K while keeping 95% of GPT-4's performance (blog ↗). Its best MT Bench router, trained on augmented data, sent 14% of calls to GPT-4 (26% when trained on Arena data alone), and the pair was GPT-4 and Mixtral 8x7B, much further apart in price than any pair above. AWS says Intelligent Prompt Routing can cut costs by up to 30%. LiteLLM benchmarked its own auto-routing on 300 prompts graded by a blinded pairwise judge: its semantic router, now deprecated, saved 58.2% and scored a 46.8% win rate against the flagship model, where 50% means parity and a randomly shuffled control scored 39.2%. Its newer tier-based complexity router did not beat that control (benchmark ↗). Every one of those numbers is measured on someone else's prompts.
What routing costs that the price table leaves out #
Four costs move the real saving below the table:
- +Prompt caching stops working across a switch. A prompt cache holds the attention state one model computed for your prefix, so no other model can use it. Google's caching API says cached content can only be used with the model it was created for, and Azure's router docs say caching benefits apply only when the same model handles consecutive requests. On Claude Sonnet 5.5, a cached read costs $0.10 per million tokens and a five-minute cache write $2.50. Twenty thousand tokens into a conversation, an agent on Sonnet pays $0.002 a turn to read that context from the cache. If the router moves the conversation to Sonnet from another model, Sonnet has never seen it, and the first turn pays $0.05 to write it, 25 times as much. Ameasured write-up on Towards Data Science ↗ found that per-request routing at the gateway, under real pricing, delivered only 13 of the 40 percentage points of saving the headline suggested, and destroyed almost twice as much cache as sticky routing.
- +A cascade pays twice for every escalation. An escalated request costs the cheap call plus the strong one, and the user waits for both. With Haiku 5.5 in front of Sonnet 5.5, a cascade that escalates 30% of requests costs 35% of always-Sonnet, against 33.5% for a router that sends the same 30% straight to Sonnet. Cascades are also only as good as their check: RouterBench found cascade performance drops rapidly as the judge's error rate rises (paper ↗ ).
- +Cheap models can take more turns per task. An agent pays per finished task, not per call. A model that needs an extra tool round or retries a malformed call can cost more per task than its list price suggests; thecheapest LLM API post covers the mechanics.
- +The router itself. A classifier call adds latency to every request, and some routers charge for it: Amazon Bedrock lists a separate charge of $1 per 1,000 requests for Intelligent Prompt Routing, twice what 1,000 calls to Haiku 5.5 cost at the call size above.
Routing is one LLM cost optimization among several, and the others are cheaper to try first. For the OpenAI and Anthropic models in the table, cached input costs 0.05x to 0.1x of the input price. OpenAI, Anthropic and Google all take 50% off batch requests for work that can wait. And if every route holds on the cheap model, you don't need a router at all; you need a migration.
Why router benchmarks don't tell you what your app loses #
Every learned or managed router decides on a prediction of quality, or a proxy for one (OpenRouter's is other users' spending), and the predictions are trained and scored on someone else's data. RouteLLM's routers were trained mainly on Chatbot Arena preference data, augmented with LLM-judge and MMLU labels, and tested on MT Bench, MMLU and GSM8K, which are chat and question-answering sets. Its "95% of GPT-4 performance" means the routed mix scored 8.8 on MT Bench against GPT-4's 9.3, which closes half of the gap between the two models, averaged over the benchmark.
Independent benchmarks are more sobering. RouterBench, built from more than 405,000 inference outcomes, found that its predictive routers did not significantly beat a baseline that mixes models in fixed proportions without reading the query. LLMRouterBench (January 2026; 21 datasets, 33 models) found that many routing methods, including OpenRouter's, the one commercial router it tested, fail to reliably beat the best single model. It reports OpenRouter at −24.7% against the best single model, and its best methods cut cost by up to 31.7% while matching that model (paper ↗). The paper tested OpenRouter's router as it was then, and the current one ranks models by spend, so treat that number as a reason to test, not a verdict on today's version.
Microsoft's own guide to evaluating its model router ↗ makes the point directly: group prompts by workload category "so that an aggregate result doesn't hide a category-specific regression". A Towards Data Science case study ↗ shows what happens when you don't. The cheap tier of a routing layer matched the strong model on 94% of a 5,000-query holdout, quality monitoring that wasn't split by tier hid the long-tail failures for months, and the team estimated the quality loss cost four to five times what routing saved.
In agents, the loss is behavior
Routing quality in an agent is mostly about what the agent does. I replayed 16 real examples from an agent on its current model and a cheaper candidate: 58.8% cheaper, 75.2% faster, not one failed or truncated call. The migration report still returned FAIL.
The cheaper model picked a different tool on 25% of comparisons against a 10% limit, searched the web where the current model answered in two sentences, and told a user an action was done without calling the tool that does it. Every call succeeded at the API level. A router that sent that agent's traffic to the cheaper model would have shipped those failures on the requests it routed there, and a dashboard of cost, latency and error rate would have called it a win. The full report is in the new model was 59% cheaper and 75% faster, and I still wouldn't ship it.
The opposite result is just as real. A customer-support agent with six tools moved to a cheaper model over 120 production examples and held every budget, including zero regressions on its security slice, at 71.2% lower cost (the passing report). Price told me nothing about which of the two would pass.
Where routing breaks an agent #
The regressions routing introduces cluster in four places:
- +Requests that look easy and aren't. "Yes, cancel the second one" is five words. A router that scores the latest message sees an easy request; the work is in the conversation behind it. In the case study above, "where is my charge from" read as a simple billing lookup and was sometimes the start of a fraud report.
- +Switching models mid-task. OpenRouter's Auto Router can choose a different model on every turn, while Requesty's auto routing pins its choice for 60 minutes per conversation. A model picked mid-task inherits another model's half-finished plan, with different tool-call habits and a cold cache.
- +The cheap model's own agent failures. It calls a tool where none was needed or skips a required lookup, claims an action it never took, picks a neighboring tool, or passes an id with a typo. Theguide to evaluating agent tool calls covers the check for each.
- +Drift with no deploy. Thresholds are tuned on last month's traffic. When the mix shifts, the share routed to the cheap model shifts with it, and managed routers change their own pools and rankings on their own schedule.
How to check an LLM router doesn't cut quality #
Six steps. The first five run before the router takes traffic; the sixth keeps it honest after.
Step 1: sample production traffic and tag it by route #
Take 100 to 300 real inputs from production, stratified by route or intent so the rare and expensive paths are in the sample, and tag each one with its route. For an agent, take whole conversations, not single messages, so a mid-conversation switch is part of what you test. A golden suite built from production traffic is the reusable version of this.
Size the routes you plan to decide on, not just the total. Zero regressions out of 20 examples still leaves a true rate of up to about 15% at 95% confidence (the rule of three, 3/n); zero out of 100 brings it down to about 3%. How many eval cases you need has the full sizing.
Step 2: write the per-route limits before the run #
For each route, a maximum regression rate, a maximum tool-selection divergence and a minimum saving that makes routing it worth the risk. Routes that move money or touch security get zero allowed regressions, zero tool divergence and zero argument drift. Deciding after you have seen the numbers is how a bad router gets approved, because a 66% saving makes almost any regression look acceptable in hindsight.
Step 3: run both models on every example #
Run the strong model and the cheap model on every example, not only on the ones the router would send cheap. With both outputs in hand you can score any routing policy offline: for each example, the router's decision picks which of the two outputs it would have served. One paired run then answers the question for every router and every threshold you are considering. RouteLLM's evaluation works the same way, scoring routers from both models' results on each query.
Compare behavior, not just text: tool calls, arguments and structured fields per example, and read every example that regressed. An LLM judge helps on open-ended answers, but know when to trust it.
Step 4: score the router's decisions, not just the cheap model #
Two numbers per route. The misroute rate is the share of requests the router sent to the cheap model on which the cheap model regressed against the strong one; that is the quality you are giving up. The missed-saving rate is the share it kept on the strong model where the cheap one would have held; that is money left on the table. A router with a low misroute rate overall and a high one on the refund route is a router you can't ship as it is.
Step 5: measure the blended cost on the same run #
From the paired run, add up what the router's choices would have cost: for each example, the measured cost of the output it would have served. Then add the router's fee per request, any escalations, and a cache write for every turn where the chosen model differs from the previous turn's. Compare the total with the always-strong baseline on the same inputs, not with the table above. If it is much smaller than the arithmetic said, cache writes and extra turns are usually why.
Step 6: route per route, and re-run when anything moves #
Send each route where it passed and keep the rest on the strong model. A static table from route to model is the simplest router there is, and the easiest to test exactly. Then re-run the check whenever the router's version, model pool or threshold changes, when either model changes version, when the prompt or tools change, and when the traffic mix shifts. Watch the share of traffic routed to the cheap model in production: when it moves without a change on your side, the router or the traffic did.
Doing this with EvalShift #
This is the comparison EvalShift runs. It records real calls with its capture SDK and promotes them
into a frozen golden suite, and every distinct tag on an example becomes a slice, so tagging by
route gives you per-route results with no extra setup. evalshift compare --from <strong-model> --to <cheap-model> replays the suite on both models, scores every pair with the evaluators you
configure (tool calls and arguments, structure, semantic similarity, an LLM judge), and measures
cost and latency from the run's own calls.
Per-route limits go in migration_policy.slices in evalshift.yaml, keyed by tag, and each one
overrides the top-level budget only where you set it:
migration_policy:
max_overall_regression_rate: 0.05
max_tool_divergence: 0.05
max_cost_increase: 0.0
slices:
refund:
max_overall_regression_rate: 0.0
max_tool_divergence: 0.0
max_tool_argument_drift: 0.0
security:
max_overall_regression_rate: 0.0
max_tool_divergence: 0.0
max_cost_increase caps a cost increase; it can't demand a saving, so read the minimum saving from
the report's cost delta. evalshift init --profile cost-reduction scaffolds budgets tighter than
the defaults. The report gives a verdict, the per-slice results and the per-example diffs, so the
routes that held become your routing table and the ones that broke stay on the strong model.
One limit: EvalShift compares two models, and it doesn't call your router. To score a learned router, take its decision for each example and read the per-example results for the examples it would have sent to the cheap model, as in Step 4. The step-by-step migration test guide shows the whole flow, and the migration policy docs list every budget.
An LLM router's saving is arithmetic you can do before you build it. What it costs in quality is something you can only measure, route by route, on your own traffic.
FAQ #
What is an LLM router? An LLM router is a layer between an application and the model APIs that chooses which model handles each request, usually sending easy requests to a cheap model and hard ones to a strong model. It can decide by fixed rules, by a learned classifier, by a platform's quality estimates, or as a cascade that tries the cheap model first and escalates.
How much does LLM routing save? At most the share of calls routed times the price gap between
the two models. At November 2026 list prices, on prompts under 100,000 tokens, moving 70% of calls
from Claude Sonnet 5.5 to Claude Haiku 5.5 saves 66.5%, and moving 70% from gpt-6.1-terra to
gpt-6-luna saves 63%. Lost prompt caching, cascade escalations, extra turns and router fees
bring the measured saving below that.
Does LLM routing reduce quality? Usually somewhat, and the loss is uneven. Routers are scored on benchmark averages, such as RouteLLM keeping 95% of GPT-4's MT Bench performance, while an application loses quality in specific routes. In agents it shows up as wrong tool calls and claimed actions that never happened, which a successful API response and fluent text can hide.
What is the difference between an LLM router and an LLM gateway? A gateway spreads requests across deployments for availability, latency, rate limits and price, and falls back when one fails. A router chooses the model by how hard each request is. Many products, including LiteLLM, do both.
How do you evaluate an LLM router? Replay 100 to 300 tagged production inputs on both the strong and the cheap model, write per-route limits before the run, and score the router's decisions against the per-example results: the misroute rate is the share of cheap-routed requests that regressed. Measure the blended cost on the same run, and re-run when the router, the models or the traffic change.
Does LLM routing break prompt caching? Switching models does. A prompt cache holds one model's state for your prefix, so the first request after a switch writes the cache again at full or higher price. A router that picks the model once per conversation, instead of per request, loses far less of it.
Is LLM routing worth it for AI agents? Often, per route. Agents resend a long, cacheable prefix every turn and fail in tool calls rather than prose, so per-request routing within a conversation is the riskiest form. Testing each route on real traffic and routing only the routes where the cheap model held keeps the saving on those routes and keeps out the failures the test can detect.
Keep reading #
- +The cheapest LLM API in 2026, and whether it is good enough for your agent — the full price table and why list price is not what an agent costs.
- +The new model was 59% cheaper and 75% faster. I still wouldn't ship it. — the failing report, evaluator by evaluator.
- +This one passed: 71% cheaper, and the agent still does the same thing — per-slice budgets on a cheap swap that held.