Most LLM cost discussions end with the same advice: don't send everything to the biggest model. Put a small, fast model in front, escalate when you need to, and watch the bill drop.
That advice is fine. The problem is what it leaves out. The moment you add a router, you've added a second model to your system: one whose job is to make a decision, whose mistakes are mostly invisible, and which almost nobody evaluates with the same care as the models it routes between.
The tutorial version of a cascade looks like this:
def answer(request):
draft = small_model(request)
if looks_ok(draft):
return draft
return large_model(request)
Three lines of logic, and every one of them hides a decision. What does looks_ok actually check? What happens when it's wrong in either direction? How long did the user wait when the first call was thrown away?
In production, those questions are the system. Here's where it tends to go wrong. (The scenarios below are illustrative, not drawn from a specific deployment.)
Escalation usually triggers on things you can detect: malformed JSON, a failed schema check, a refusal, a low self-reported confidence score. What it rarely catches is a well-formed, fluent, wrong answer.
Picture a support assistant that routes "simple" billing questions to the small model. A customer asks about a proration edge case. The small model returns a clean, polite, perfectly structured answer that quotes the wrong number. Every validator passes. The cascade reports a successful cheap resolution. Your dashboard says routing is working.
The failures your router can't see are the ones that cost the most.
Many routers decide based on a proxy for difficulty: prompt length, keyword lists, a classifier trained on "easy vs. hard" labels someone made up in an afternoon.
But difficulty is relative to the model. A short prompt asking for a regex that handles a nasty edge case is hard. A long prompt asking to summarize a meeting transcript might be easy. If your routing signal is a vague human intuition about difficulty, you're encoding that intuition as a production decision without ever measuring it.
A cascade that escalates often pays twice: one call to the small model, then a full call to the large one. If your escalation rate on some task type is high, the "fast first pass" is adding latency, not removing it.
Average latency hides this. Look at end-to-end latency per task type, including retries and validation time, and the picture can flip: for some categories, going straight to the larger model is both faster and simpler.
A wrong answer in an internal draft generator costs someone thirty seconds. A wrong answer in a compliance summary or a code change that ships can cost far more. A router optimized purely on token spend will happily save cents on requests where an error costs real money.
Evaluate the router as its own component, against real traffic, broken down by task type:
per task type:
- quality of the small-model path vs. the large-model path
- escalation rate
- false "ok" rate (validator passed, answer was wrong)
- p50 / p95 end-to-end latency, including retries
- cost per request, and estimated cost of an error
The false "ok" rate is the one most teams don't have, because it requires sampling routed answers and grading them. It's also the one that tells you whether your cheap path is actually cheap.
A practical starting point:
Model routing isn't a cost optimization with a classifier attached. It's a decision system, and it deserves the same evaluation discipline as any model you'd put in front of users.
How are you deciding where your cheap path ends? Measured boundaries, or gut feel that hasn't been revisited?