{"slug": "my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly", "title": "My LLM Router Spent 8x More and Matched the Cheap Model Exactly", "summary": "A developer tested a confidence-based LLM routing strategy using llama3.2:1b as the cheap tier and llama3.1:8b as the strong tier across 30 programmatically graded tasks, and found the router escalated 14 of 30 tasks yet matched always-cheap accuracy exactly at 43.3% while costing 8.2 times more. Per-task inspection revealed two opposing effects that canceled out in this run: self-reported confidence was a poor predictor of correctness, and requesting a confidence score changed the answers themselves. The write-up contrasts this untrained approach with RouteLLM, which reports over 85% cost reduction on MT Bench while retaining 95% of GPT-4 quality.", "body_md": "I built the LLM model routing strategy almost everyone reaches for first: ask the cheap model to\n\nanswer, have it report how confident it is, and send the question to a stronger model whenever\n\nthat confidence drops below a threshold. Thirty tasks, three strategies, real local models, real\n\npublished pricing applied to the real token counts. The router escalated 14 of the 30 tasks to\n\nthe strong model and ended up with exactly the same accuracy as never routing at all, 43.3%\n\neither way, while spending 8.2 times more money to get there.\n\nMy first read of that number was \"routing did nothing here.\" That read was wrong, and the only\n\nreason I know it's wrong is that I went back and checked every task individually instead of\n\nstopping at the headline. Two real, opposite effects happened. They canceled out almost exactly,\n\nwhich is a coincidence of this specific run, not a law the router obeys.\n\n**Proof block.** The setup, before any of the findings:\n\n`llama3.2:1b` (cheap tier) and `llama3.1:8b` (strong tier), via local Ollama 0.34.2.\n**TL;DR.** What actually came out of checking the numbers instead of trusting the headline:\n\n**The question this is actually testing:** does a model's own self-reported confidence predict\n\nwhether its answer is correct, well enough to use as a routing signal? A falsifiable claim, not a\n\nvibe. A separate, closer question the write-up below surfaces almost by accident: does asking for\n\nthat confidence score change the answer itself?\n\nLLM model routing means sending each request to the cheapest model that can actually handle it,\n\nand only paying for a stronger model when the task needs it. The published version that works is\n\n[RouteLLM](https://lmsys.org/blog/2024-07-01-routellm/) (Ong et al., 2024, UC Berkeley/LMSYS): a router trained on real preference data, human and\n\nLLM-judge comparisons of which model actually answered better, achieving over 85% cost reduction\n\non MT Bench while keeping 95% of GPT-4's quality, at best sending only 14% of traffic to the\n\nstrong model (GPT-4 Turbo vs. Mixtral 8x7B, their best augmented-data router).\n\nThat takes a labeled dataset and a training step most people building an agent on a Tuesday don't\n\nhave. The version that's actually quick to try is the one in this article: skip the training,\n\njust ask the cheap model how sure it is, and escalate when it says it isn't.\n\n**The 30 tasks** span the kind of decisions a production agent harness routes constantly: factual\n\nlookups and unit conversion (easy), log-field extraction and multi-step cost arithmetic (medium),\n\nand multi-constraint reasoning with a distractor built in (hard). Every task has a programmatic\n\n`check()` function, exact or substring match against a known answer, so grading never depends on\n\nmy own judgment call.\n\n**The routing rule under test:**\n\n``` python\nESCALATE_BELOW = 70\n\ndef route(task, cheap_model, strong_model):\n    routed_prompt = task.prompt + (\n        \"\\n\\nAfter your answer, on a new line write exactly: \"\n        \"CONFIDENCE: <a number 0-100 for how sure you are>.\"\n    )\n    answer = call(cheap_model, routed_prompt)\n    confidence = parse_confidence(answer)\n\n    if confidence is None or confidence < ESCALATE_BELOW:\n        return call(strong_model, task.prompt)  # escalate, plain prompt\n    return strip_confidence_line(answer)\n```\n\nCost for an escalated task is the cheap call's cost plus the strong call's cost. Cost for a\n\nnon-escalated task is just the cheap call. I ran all three conditions, always-cheap (plain\n\nprompt, no confidence instruction), always-strong (plain prompt), and routed, against every one\n\nof the 30 tasks.\n\n*Image by author. Pricing: DeepSeek V4.1 Flash ($0.30/$1.20 per M tokens) for the cheap tier, GPT-5.6 Sol ($4/$20 per M tokens) for the strong tier, applied to real local token counts.*\n\n| Strategy | Accuracy | Cost (30 tasks) | \n|---|---|---|\n| Always cheap | 43.3% (13/30) | $0.0024 | \n| Always strong | 66.7% (20/30) | $0.0622 | \n| Routed | 43.3% (13/30) | $0.0195 | \n\nThe router escalated 14 of 30 tasks (46.7%). Here is what those 14 escalations actually bought:\n\nSo escalation itself was mildly useful on net: +3 correct answers, and it never broke an answer\n\nthat was already right (confirmed, all 8 wasted-bucket tasks stayed correct on the strong model\n\ntoo). If that were the whole story, routed accuracy should have landed at 16/30, not 13/30. It\n\ndidn't. Something else happened.\n\nThe 16 tasks that did not escalate were not simply \"kept the cheap model's plain answer.\" Each\n\none used a second call: the task prompt plus the added confidence instruction. I'd assumed, going\n\nin, that this second call would answer the task the same way the plain call did, it's the same\n\nmodel, same temperature, same seed, just one more line of instruction asking it to also grade\n\nitself. That assumption was wrong on 5 of those 16 tasks.\n\n*Image by author, computed directly from raw_results.jsonl.*\n\nFour tasks that the plain cheap model answered correctly got answered **wrong** once the\n\nconfidence instruction was attached to the same prompt. One task flipped the other way, wrong\n\nwithout the instruction, right with it. Net: minus three correct answers among the non-escalated\n\nset, purely from adding one line asking the model to grade its own certainty. Plain-prompt\n\naccuracy on those exact 16 tasks would have been 31.3% (5/16); the actual routed-prompt accuracy\n\non them was 12.5% (2/16).\n\nOne of the four right-to-wrong flips was the simplest task in the whole set: \"what is 47 + 58?\"\n\nAsked plainly, the cheap model answered 105, correct. Asked the identical question with one extra\n\nsentence attached, report a confidence score afterward, it answered 95, wrong, and reported 80%\n\nconfidence in it. Nothing about the math changed. The only difference between the two calls was\n\none trailing instruction the model never even needed to act on until after it had already\n\nanswered. For a 1.2-billion-parameter model, that was apparently enough to disturb the arithmetic\n\nitself, not just the metadata wrapped around it.\n\nA second, costlier case, task h5, shows the same failure mode on harder math: \"a classifier has\n\n95% accuracy on 80% of traffic and 60% accuracy on the other 20%, what's the overall accuracy?\"\n\nThe correct answer is 88%. The cheap model wrote out the right formula, `(0.8 x 95%) + (0.2 x`, under the routed prompt, then slipped on the arithmetic and wrote 76.8% instead of 88%,\n\n60%)\n\nreporting **100% confidence** in that wrong number. This one wasn't a flip (the plain-prompt\n\ncheap model missed it too), but it's the sharpest example in the set of what \"confidently wrong\"\n\nactually looks like: a correct method, a blown calculation, and total certainty in the output.\n\nI did not design this run to isolate that effect, which means I can't yet say how general it is\n\nacross model sizes or instruction phrasing. What I can say is that it happened, on a clean,\n\ndeterministic, reproducible setup, and it happened to land in the opposite direction of the\n\nescalation gain, which is the only reason the headline number looked like nothing changed. A\n\nrouter implementation detail, whether you ask for a confidence score inline versus in a separate\n\ncall, measurably changed what was being routed, before the routing decision even mattered. If you\n\nare building something like this, that is worth a dedicated check of its own, not an assumption.\n\n**Self-reported confidence is not a usable routing signal by default.** Here it failed in three\n\nseparate ways at once: wasting money re-asking questions it already had right, silently keeping\n\nwrong answers it was sure about (14 of 16 non-escalated tasks), and degrading accuracy as a side\n\neffect of the instruction used to elicit it. Any one of those would be worth fixing before\n\nshipping a router built this way. Getting all three at once is a strong argument against the\n\none-line version of this idea, not a reason to abandon routing altogether.\n\n**Test the full implementation, not just the escalation logic.** The part I expected to be weak\n\nhere (confidence threshold picking) was weak, but not the only problem. The part I didn't expect\n\nto matter (how the confidence instruction is delivered) cost as much accuracy as the escalation\n\nlogic gained back. If I'd only checked the top-line numbers, I would have reported \"routing had no\n\neffect\" and missed both real findings underneath it.\n\n**If routing cost actually matters for your workload, build toward RouteLLM's approach, not this one.** A router trained on real preference comparisons between model outputs is the proven path.\n\nEverything here, the 30 tasks, the exact prompts, every raw model response, the token counts and\n\ncost math, the three annotated failure cases, and the chart generator, is public and reproducible:\n\n**[github.com/priyansh18/confidence-router-benchmark](https://github.com/priyansh18/confidence-router-benchmark)**. Deterministic by design; a clean re-run\n\nreproduced every accuracy number exactly when I checked it mid-project, which is what first made\n\nthe discrepancy in this article worth chasing down instead of writing off as noise.\n\nIf you're building agent infrastructure and want a second pair of eyes on a cost or reliability\n\nproblem before it ships, I take on a limited number of consulting calls through\n\n[Topmate](https://topmate.io/priyansh_singhal10).\n\n*Parts of this article's drafting were assisted by an AI writing tool; the experiment, the data, the repo, and the analysis are my own.*\n\n**One real question, not a generic one:** if you've shipped a confidence-based or heuristic\n\nrouter in production, did you ever check whether the signal you used to decide \"escalate or not\"\n\nactually correlated with correctness on your own traffic, or did it just look reasonable at\n\ndesign time the way mine did before I ran this?", "url": "https://wpnews.pro/news/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly", "canonical_source": "https://dev.to/priyansh_singhal_5975e7d3/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly-18d9", "published_at": "2026-10-21 12:32:03+00:00", "updated_at": "2026-10-21 12:40:15.218313+00:00", "lang": "en", "topics": ["large-language-models", "ai-agents", "ai-tools", "machine-learning"], "entities": ["Ollama", "llama3.2:1b", "llama3.1:8b", "RouteLLM", "UC Berkeley", "LMSYS", "DeepSeek", "GPT-5.6 Sol"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly", "markdown": "https://wpnews.pro/news/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly.md", "text": "https://wpnews.pro/news/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly.txt", "jsonld": "https://wpnews.pro/news/my-llm-router-spent-8x-more-and-matched-the-cheap-model-exactly.jsonld"}}