I built the LLM model routing strategy almost everyone reaches for first: ask the cheap model to
answer, have it report how confident it is, and send the question to a stronger model whenever
that confidence drops below a threshold. Thirty tasks, three strategies, real local models, real
published pricing applied to the real token counts. The router escalated 14 of the 30 tasks to
the strong model and ended up with exactly the same accuracy as never routing at all, 43.3%
either way, while spending 8.2 times more money to get there.
My first read of that number was "routing did nothing here." That read was wrong, and the only
reason I know it's wrong is that I went back and checked every task individually instead of
stopping at the headline. Two real, opposite effects happened. They canceled out almost exactly,
which is a coincidence of this specific run, not a law the router obeys.
Proof block. The setup, before any of the findings:
llama3.2:1b (cheap tier) and llama3.1:8b (strong tier), via local Ollama 0.34.2.
TL;DR. What actually came out of checking the numbers instead of trusting the headline:
The question this is actually testing: does a model's own self-reported confidence predict
whether its answer is correct, well enough to use as a routing signal? A falsifiable claim, not a
vibe. A separate, closer question the write-up below surfaces almost by accident: does asking for
that confidence score change the answer itself?
LLM model routing means sending each request to the cheapest model that can actually handle it,
and only paying for a stronger model when the task needs it. The published version that works is
RouteLLM (Ong et al., 2024, UC Berkeley/LMSYS): a router trained on real preference data, human and
LLM-judge comparisons of which model actually answered better, achieving over 85% cost reduction
on MT Bench while keeping 95% of GPT-4's quality, at best sending only 14% of traffic to the
strong model (GPT-4 Turbo vs. Mixtral 8x7B, their best augmented-data router).
That takes a labeled dataset and a training step most people building an agent on a Tuesday don't
have. The version that's actually quick to try is the one in this article: skip the training,
just ask the cheap model how sure it is, and escalate when it says it isn't.
The 30 tasks span the kind of decisions a production agent harness routes constantly: factual
lookups and unit conversion (easy), log-field extraction and multi-step cost arithmetic (medium),
and multi-constraint reasoning with a distractor built in (hard). Every task has a programmatic
check() function, exact or substring match against a known answer, so grading never depends on
my own judgment call.
The routing rule under test:
ESCALATE_BELOW = 70
def route(task, cheap_model, strong_model):
routed_prompt = task.prompt + (
"\n\nAfter your answer, on a new line write exactly: "
"CONFIDENCE: <a number 0-100 for how sure you are>."
)
answer = call(cheap_model, routed_prompt)
confidence = parse_confidence(answer)
if confidence is None or confidence < ESCALATE_BELOW:
return call(strong_model, task.prompt) # escalate, plain prompt
return strip_confidence_line(answer)
Cost for an escalated task is the cheap call's cost plus the strong call's cost. Cost for a
non-escalated task is just the cheap call. I ran all three conditions, always-cheap (plain
prompt, no confidence instruction), always-strong (plain prompt), and routed, against every one
of the 30 tasks.
Image by author. Pricing: DeepSeek V4.1 Flash ($0.30/$1.20 per M tokens) for the cheap tier, GPT-5.6 Sol ($4/$20 per M tokens) for the strong tier, applied to real local token counts.
| Strategy | Accuracy | Cost (30 tasks) |
|---|---|---|
| Always cheap | 43.3% (13/30) | $0.0024 |
| Always strong | 66.7% (20/30) | $0.0622 |
| Routed | 43.3% (13/30) | $0.0195 |
The router escalated 14 of 30 tasks (46.7%). Here is what those 14 escalations actually bought:
So escalation itself was mildly useful on net: +3 correct answers, and it never broke an answer
that was already right (confirmed, all 8 wasted-bucket tasks stayed correct on the strong model
too). If that were the whole story, routed accuracy should have landed at 16/30, not 13/30. It
didn't. Something else happened.
The 16 tasks that did not escalate were not simply "kept the cheap model's plain answer." Each
one used a second call: the task prompt plus the added confidence instruction. I'd assumed, going
in, that this second call would answer the task the same way the plain call did, it's the same
model, same temperature, same seed, just one more line of instruction asking it to also grade
itself. That assumption was wrong on 5 of those 16 tasks.
Image by author, computed directly from raw_results.jsonl.
Four tasks that the plain cheap model answered correctly got answered wrong once the
confidence instruction was attached to the same prompt. One task flipped the other way, wrong
without the instruction, right with it. Net: minus three correct answers among the non-escalated
set, purely from adding one line asking the model to grade its own certainty. Plain-prompt
accuracy on those exact 16 tasks would have been 31.3% (5/16); the actual routed-prompt accuracy
on them was 12.5% (2/16).
One of the four right-to-wrong flips was the simplest task in the whole set: "what is 47 + 58?"
Asked plainly, the cheap model answered 105, correct. Asked the identical question with one extra
sentence attached, report a confidence score afterward, it answered 95, wrong, and reported 80%
confidence in it. Nothing about the math changed. The only difference between the two calls was
one trailing instruction the model never even needed to act on until after it had already
answered. For a 1.2-billion-parameter model, that was apparently enough to disturb the arithmetic
itself, not just the metadata wrapped around it.
A second, costlier case, task h5, shows the same failure mode on harder math: "a classifier has
95% accuracy on 80% of traffic and 60% accuracy on the other 20%, what's the overall accuracy?"
The correct answer is 88%. The cheap model wrote out the right formula, (0.8 x 95%) + (0.2 x, under the routed prompt, then slipped on the arithmetic and wrote 76.8% instead of 88%,
60%)
reporting 100% confidence in that wrong number. This one wasn't a flip (the plain-prompt
cheap model missed it too), but it's the sharpest example in the set of what "confidently wrong"
actually looks like: a correct method, a blown calculation, and total certainty in the output.
I did not design this run to isolate that effect, which means I can't yet say how general it is
across model sizes or instruction phrasing. What I can say is that it happened, on a clean,
deterministic, reproducible setup, and it happened to land in the opposite direction of the
escalation gain, which is the only reason the headline number looked like nothing changed. A
router implementation detail, whether you ask for a confidence score inline versus in a separate
call, measurably changed what was being routed, before the routing decision even mattered. If you
are building something like this, that is worth a dedicated check of its own, not an assumption.
Self-reported confidence is not a usable routing signal by default. Here it failed in three
separate ways at once: wasting money re-asking questions it already had right, silently keeping
wrong answers it was sure about (14 of 16 non-escalated tasks), and degrading accuracy as a side
effect of the instruction used to elicit it. Any one of those would be worth fixing before
shipping a router built this way. Getting all three at once is a strong argument against the
one-line version of this idea, not a reason to abandon routing altogether.
Test the full implementation, not just the escalation logic. The part I expected to be weak
here (confidence threshold picking) was weak, but not the only problem. The part I didn't expect
to matter (how the confidence instruction is delivered) cost as much accuracy as the escalation
logic gained back. If I'd only checked the top-line numbers, I would have reported "routing had no
effect" and missed both real findings underneath it.
If routing cost actually matters for your workload, build toward RouteLLM's approach, not this one. A router trained on real preference comparisons between model outputs is the proven path.
Everything here, the 30 tasks, the exact prompts, every raw model response, the token counts and
cost math, the three annotated failure cases, and the chart generator, is public and reproducible:
github.com/priyansh18/confidence-router-benchmark. Deterministic by design; a clean re-run
reproduced every accuracy number exactly when I checked it mid-project, which is what first made
the discrepancy in this article worth chasing down instead of writing off as noise.
If you're building agent infrastructure and want a second pair of eyes on a cost or reliability
problem before it ships, I take on a limited number of consulting calls through
Parts of this article's drafting were assisted by an AI writing tool; the experiment, the data, the repo, and the analysis are my own.
One real question, not a generic one: if you've shipped a confidence-based or heuristic
router in production, did you ever check whether the signal you used to decide "escalate or not"
actually correlated with correctness on your own traffic, or did it just look reasonable at
design time the way mine did before I ran this?