My LLM Router Spent 8x More and Matched the Cheap Model Exactly

A developer tested a confidence-based LLM routing strategy using llama3.2:1b as the cheap tier and llama3.1:8b as the strong tier across 30 programmatically graded tasks, and found the router escalated 14 of 30 tasks yet matched always-cheap accuracy exactly at 43.3% while costing 8.2 times more. Per-task inspection revealed two opposing effects that canceled out in this run: self-reported confidence was a poor predictor of correctness, and requesting a confidence score changed the answers themselves. The write-up contrasts this untrained approach with RouteLLM, which reports over 85% cost reduction on MT Bench while retaining 95% of GPT-4 quality.

I built the LLM model routing strategy almost everyone reaches for first: ask the cheap model to answer, have it report how confident it is, and send the question to a stronger model whenever that confidence drops below a threshold. Thirty tasks, three strategies, real local models, real published pricing applied to the real token counts. The router escalated 14 of the 30 tasks to the strong model and ended up with exactly the same accuracy as never routing at all, 43.3% either way, while spending 8.2 times more money to get there. My first read of that number was "routing did nothing here." That read was wrong, and the only reason I know it's wrong is that I went back and checked every task individually instead of stopping at the headline. Two real, opposite effects happened. They canceled out almost exactly, which is a coincidence of this specific run, not a law the router obeys. Proof block. The setup, before any of the findings: llama3.2:1b cheap tier and llama3.1:8b strong tier , via local Ollama 0.34.2. TL;DR. What actually came out of checking the numbers instead of trusting the headline: The question this is actually testing: does a model's own self-reported confidence predict whether its answer is correct, well enough to use as a routing signal? A falsifiable claim, not a vibe. A separate, closer question the write-up below surfaces almost by accident: does asking for that confidence score change the answer itself? LLM model routing means sending each request to the cheapest model that can actually handle it, and only paying for a stronger model when the task needs it. The published version that works is RouteLLM https://lmsys.org/blog/2024-07-01-routellm/ Ong et al., 2024, UC Berkeley/LMSYS : a router trained on real preference data, human and LLM-judge comparisons of which model actually answered better, achieving over 85% cost reduction on MT Bench while keeping 95% of GPT-4's quality, at best sending only 14% of traffic to the strong model GPT-4 Turbo vs. Mixtral 8x7B, their best augmented-data router . That takes a labeled dataset and a training step most people building an agent on a Tuesday don't have. The version that's actually quick to try is the one in this article: skip the training, just ask the cheap model how sure it is, and escalate when it says it isn't. The 30 tasks span the kind of decisions a production agent harness routes constantly: factual lookups and unit conversion easy , log-field extraction and multi-step cost arithmetic medium , and multi-constraint reasoning with a distractor built in hard . Every task has a programmatic check function, exact or substring match against a known answer, so grading never depends on my own judgment call. The routing rule under test: python ESCALATE BELOW = 70 def route task, cheap model, strong model : routed prompt = task.prompt + "\n\nAfter your answer, on a new line write exactly: " "CONFIDENCE: <a number 0-100 for how sure you are ." answer = call cheap model, routed prompt confidence = parse confidence answer if confidence is None or confidence < ESCALATE BELOW: return call strong model, task.prompt escalate, plain prompt return strip confidence line answer Cost for an escalated task is the cheap call's cost plus the strong call's cost. Cost for a non-escalated task is just the cheap call. I ran all three conditions, always-cheap plain prompt, no confidence instruction , always-strong plain prompt , and routed, against every one of the 30 tasks. Image by author. Pricing: DeepSeek V4.1 Flash $0.30/$1.20 per M tokens for the cheap tier, GPT-5.6 Sol $4/$20 per M tokens for the strong tier, applied to real local token counts. | Strategy | Accuracy | Cost 30 tasks | |---|---|---| | Always cheap | 43.3% 13/30 | $0.0024 | | Always strong | 66.7% 20/30 | $0.0622 | | Routed | 43.3% 13/30 | $0.0195 | The router escalated 14 of 30 tasks 46.7% . Here is what those 14 escalations actually bought: So escalation itself was mildly useful on net: +3 correct answers, and it never broke an answer that was already right confirmed, all 8 wasted-bucket tasks stayed correct on the strong model too . If that were the whole story, routed accuracy should have landed at 16/30, not 13/30. It didn't. Something else happened. The 16 tasks that did not escalate were not simply "kept the cheap model's plain answer." Each one used a second call: the task prompt plus the added confidence instruction. I'd assumed, going in, that this second call would answer the task the same way the plain call did, it's the same model, same temperature, same seed, just one more line of instruction asking it to also grade itself. That assumption was wrong on 5 of those 16 tasks. Image by author, computed directly from raw results.jsonl. Four tasks that the plain cheap model answered correctly got answered wrong once the confidence instruction was attached to the same prompt. One task flipped the other way, wrong without the instruction, right with it. Net: minus three correct answers among the non-escalated set, purely from adding one line asking the model to grade its own certainty. Plain-prompt accuracy on those exact 16 tasks would have been 31.3% 5/16 ; the actual routed-prompt accuracy on them was 12.5% 2/16 . One of the four right-to-wrong flips was the simplest task in the whole set: "what is 47 + 58?" Asked plainly, the cheap model answered 105, correct. Asked the identical question with one extra sentence attached, report a confidence score afterward, it answered 95, wrong, and reported 80% confidence in it. Nothing about the math changed. The only difference between the two calls was one trailing instruction the model never even needed to act on until after it had already answered. For a 1.2-billion-parameter model, that was apparently enough to disturb the arithmetic itself, not just the metadata wrapped around it. A second, costlier case, task h5, shows the same failure mode on harder math: "a classifier has 95% accuracy on 80% of traffic and 60% accuracy on the other 20%, what's the overall accuracy?" The correct answer is 88%. The cheap model wrote out the right formula, 0.8 x 95% + 0.2 x , under the routed prompt, then slipped on the arithmetic and wrote 76.8% instead of 88%, 60% reporting 100% confidence in that wrong number. This one wasn't a flip the plain-prompt cheap model missed it too , but it's the sharpest example in the set of what "confidently wrong" actually looks like: a correct method, a blown calculation, and total certainty in the output. I did not design this run to isolate that effect, which means I can't yet say how general it is across model sizes or instruction phrasing. What I can say is that it happened, on a clean, deterministic, reproducible setup, and it happened to land in the opposite direction of the escalation gain, which is the only reason the headline number looked like nothing changed. A router implementation detail, whether you ask for a confidence score inline versus in a separate call, measurably changed what was being routed, before the routing decision even mattered. If you are building something like this, that is worth a dedicated check of its own, not an assumption. Self-reported confidence is not a usable routing signal by default. Here it failed in three separate ways at once: wasting money re-asking questions it already had right, silently keeping wrong answers it was sure about 14 of 16 non-escalated tasks , and degrading accuracy as a side effect of the instruction used to elicit it. Any one of those would be worth fixing before shipping a router built this way. Getting all three at once is a strong argument against the one-line version of this idea, not a reason to abandon routing altogether. Test the full implementation, not just the escalation logic. The part I expected to be weak here confidence threshold picking was weak, but not the only problem. The part I didn't expect to matter how the confidence instruction is delivered cost as much accuracy as the escalation logic gained back. If I'd only checked the top-line numbers, I would have reported "routing had no effect" and missed both real findings underneath it. If routing cost actually matters for your workload, build toward RouteLLM's approach, not this one. A router trained on real preference comparisons between model outputs is the proven path. Everything here, the 30 tasks, the exact prompts, every raw model response, the token counts and cost math, the three annotated failure cases, and the chart generator, is public and reproducible: github.com/priyansh18/confidence-router-benchmark https://github.com/priyansh18/confidence-router-benchmark . Deterministic by design; a clean re-run reproduced every accuracy number exactly when I checked it mid-project, which is what first made the discrepancy in this article worth chasing down instead of writing off as noise. If you're building agent infrastructure and want a second pair of eyes on a cost or reliability problem before it ships, I take on a limited number of consulting calls through Topmate https://topmate.io/priyansh singhal10 . Parts of this article's drafting were assisted by an AI writing tool; the experiment, the data, the repo, and the analysis are my own. One real question, not a generic one: if you've shipped a confidence-based or heuristic router in production, did you ever check whether the signal you used to decide "escalate or not" actually correlated with correctness on your own traffic, or did it just look reasonable at design time the way mine did before I ran this?