Last week I argued the obvious part: most production LLM traffic — extraction, classification, short rewrites — rarely needs the frontier model, and routing it to cheaper models (Chinese open-weight models are typically 70%+ cheaper, often up to 90%+ on China models) turns a flat bill into a blended one.
The easy part is done. The hard part is the question nobody answers: did the routing actually work, or did you just make your bills cheaper and your outputs silently worse?
The first metric teams reach for is "did the call succeed." 200 OK. JSON parsed. Task completed. Green checkmark.
That metric can't answer the question. A cheap-model call can return 200, parse cleanly, and still be wrong in a way that only surfaces three steps later. The call that looks fine and is quietly wrong is the dangerous failure — and it's exactly the failure routing introduces, because you changed which model handles the work.
For a sample of real traffic on each route, compare the distribution of outputs against a frontier baseline — not a single "did it pass" flag. A median output length collapsing from ~20k tokens to 0 is a different signal than an empty result. A confidence score shifting by a point is noise; a whole cluster of outputs landing in a range the frontier never produced is a behavior change. You only see this if you measure the blend, not the pass rate. Route cheap, but verify the cheap route is still producing output you'd have trusted before.
The second trap is counting tokens. "I saved 70% per call" is true and irrelevant if the cheap call fails 15% of the time and each failure triggers a retry or an escalation to frontier anyway.
The number that matters is cost per successful task: total spend across the retry and escalation path divided by tasks that actually completed correctly. A cheap-first policy with a clean fallback can still win — but you have to measure the whole path, not the first hop. If your "savings" vanish once you include the retries, you haven't optimized, you've deferred the cost.
Here's the one people miss: when you change the routing policy and re-run your eval, you're not measuring the model. You're measuring the router plus the model. Version the harness — the prompts, the retry logic, the memory, the tool schemas — the same way you version the model. A benchmark number with an unversioned harness is a number you can't reproduce and can't trust.
And don't trust a single run. "All 20 passed" tells you the mean passed; it tells you nothing about the tail. Variance in latency, in failure modes, in output quality lives in the tail, and the tail is where production breaks. One trial is a point estimate, not a confidence interval.
This is where it lands hardest for teams in Southeast Asia — Malaysia, Singapore, Indonesia — building on tight infrastructure budgets and answering to data-sovereignty rules (Malaysia's PDPA, for instance) that US-default tooling tends to ignore. When every dollar of inference is scrutinized, "we saved 70%" on a slide means nothing if nobody measured whether the work still got done. A routing gateway that is SG-hosted (Tencent Cloud) and PDPA-aligned lets a KL-based team route cheap-when-possible and frontier-when-needed — but the saving only counts if you can prove the cheap route is trustworthy.
Stop celebrating "everything succeeded." Start measuring the blend. Pick a sample, compare distributions per route, compute cost per successful task including retries, and version the harness so the number means something next month. Routing without measurement is just hope with extra steps.
If you're wiring this up, TokenLat exposes a single OpenAI-compatible endpoint over 25+ models with request-level tracing, so the blend you measure is the blend you actually shipped.