On July 30, OpenAI cut GPT-5.6 Luna to $0.20 per million input tokens and $1.20 per million output — down from $1 and $6. An 80% cut. Azure matched it on August 1.
We did what most teams did with that news: nothing. Our gateway sent every request to the strong model, because it always had, and because "just use the cheap model" sounds like a decision that comes back as a support ticket three weeks later.
Then someone put the bill next to the traffic mix, and the awkward part was obvious. The overwhelming majority of our requests were title this thread, summarise this diff, extract the fields from this form, name this file. We were paying frontier prices to generate three-word document titles.
Here's what we shipped instead. It's about forty lines of Go, it moved 81% of requests off the expensive model, and it cut the bill by 71%.
It also broke four things, and those are the interesting part.
The obvious design — the one everyone writes first — is a classifier in front of the router:
Look at the incoming request. Decide if it's
easyorhard. Send it to the model that matches.
We built this. It's worse than it looks, for three reasons.
You need a model to run the classifier. Now every request pays an extra call before any work happens. A cheap classifier is another Luna call and another 300ms; an accurate classifier is a strong-model call, which is the cost you were trying to avoid.
Short is not easy. "Fix the timezone bug" is eleven characters and needs everything you've got. "Summarise the following 4,000-word RFC" is long and trivial. Length, token count, keyword lists — every cheap heuristic we tried correlated with the shape of the request and not with the difficulty of it.
You're predicting the answer before you've seen it. That's the actual problem. Difficulty is a property of the work, and the only honest way to learn it is to do the work.
The rule:don't classify the prompt. Run the cheap model and judge the output.
The design that works is embarrassingly simple:
The thing that makes this viable is arithmetic that surprised us. At our average request shape — roughly 3,000 input tokens and 700 output — a Luna call costs about $0.0014 and a strong-model call about $0.0145. Ten to one.
So a request that gets escalated costs 0.0014 + 0.0145 = $0.0159
instead of $0.0145
. About 10% more.
Work out the break-even escalation rate:
cheap + (p × strong) < strong
p < (strong − cheap) / strong
p < (0.0145 − 0.0014) / 0.0145
p < 0.90
You would have to escalate nine times out of ten before the cheap attempt costs you money. We escalate 15% of the time. Cost is not the constraint here, and if you're arguing about whether cheap-first is worth the double billing, you're arguing about the wrong resource.
Latency is the constraint. We'll come back to that.
type Result struct {
Text string
Model string
Tokens Usage
}
// A gate returns the reason this answer can't be trusted, or "" to accept it.
type Gate func(req Request, out string, fin FinishReason) string
func (r *Router) Do(ctx context.Context, req Request) (Result, error) {
if req.ForceStrong || r.alwaysStrong[req.Kind] {
return r.call(ctx, r.strong, req)
}
cheap, err := r.call(ctx, r.cheap, req)
if err != nil {
// Availability, not quality. Fall through, don't fail the request.
return r.call(ctx, r.strong, req)
}
for _, gate := range r.gates {
reason := gate(req, cheap.Text, cheap.Finish)
if reason == "" {
continue
}
r.metrics.Escalate(req.Kind, reason)
strong, err := r.call(ctx, r.strong, req) // NB: req, not req+cheap.Text
if err != nil {
return cheap, nil // degraded, not failed
}
return strong, nil
}
r.metrics.Accept(req.Kind)
return cheap, nil
}
Two lines in there are load-bearing and neither is obvious.
return r.call(ctx, r.strong, req)
when the cheap call errors — a 429 or a timeout is an availability problem, and availability problems should not surface to the user as a failed request when you have a second provider sitting right there.
return cheap, nil
when the strong call errors — you already have an answer. It may be a worse answer. Shipping a worse answer beats shipping a spinner.
And the comment on the escalation call is the one people get wrong, which is failure mode #4 below.
The gates are the whole product. The router is plumbing.
var defaultGates = []Gate{
SchemaInvalid, // had to be JSON matching a schema, and wasn't
ToolArgsMissing, // called a tool, omitted a required argument
Truncated, // finish reason != stop
EmptyOrHedged, // under 24 chars, or matches the hedge set
}
They are ordered by how cheap they are to evaluate, and every one of them is structural. None of them asks a model to grade another model.
** SchemaInvalid** does the most work by a distance. Anything with a defined output shape — field extraction, classification, structured summaries — gets validated against the schema you already have. If it doesn't parse or doesn't conform, escalate. This gate alone catches about 60% of our escalations.
** ToolArgsMissing** is the same idea for function calls. The cheap model picks the right tool far more reliably than it fills in the right arguments, and a missing required argument is a free, exact signal.
** Truncated** is one field comparison and people skip it constantly. A
finish_reason
of length
means you have a sentence that stops mid-** EmptyOrHedged** is the weakest one, and I want to be specific about how weak, because it's the one everybody wants to build first.
We started with the intuitive version: ask the cheap model to say when it isn't confident, then escalate on that. It fired on 0.4% of responses. Our measured error rate on the same traffic was around 15%. The model's self-reported uncertainty was not a signal, it was decoration.
What actually works in that slot is a small, boring list: empty, under 24 characters, or an exact-ish match against a hedge set you build by reading two hundred real failures ("I don't have enough information"
, "As an AI"
, "Could you clarify"
). Not confidence. Refusal.
The rule:gate on structure you can check, not on the model's opinion of itself.
This was immediate and it's the real cost of the design.
| All-strong | Cheap-first + gate | |
|---|---|---|
| p50 | 2.9s | 1.4s |
| p95 | 7.8s | 9.6s |
| p99 | 11.2s | 16.4s |
Luna is fast, so the 85% that get accepted got much faster. The 15% that escalate pay for both calls, serially, and they land in your tail.
If you have a latency SLO, that tail is where the design either survives or doesn't. Two things helped: run the gates on the streamed head rather than the finished response where you can, and put a hard escalateBudget
on the clock — if the cheap call already burned 4 seconds, return it and log the miss rather than starting a second call you can't afford.
The whole architecture assumes you get to look at the output before deciding. Streaming assumes you've already committed.
There is no clever fix, only a choice:
Most of the money was in the first bucket anyway. Structured background work is high volume and nobody is watching a cursor blink at it.
The output is fluent. It's well-formatted. It uses your headings, it hits your tone, it's the right length. It's just wrong.
That's why "does this look like a good answer" gates — including LLM-as-judge in the hot path — did badly for us. Fluency is exactly the axis where the price gap has closed most. Judgement, multi-step reasoning, and knowing what it doesn't know are where it hasn't closed at all.
Structural gates work because they don't have an opinion. Valid JSON is valid JSON.
Our first version passed the failed attempt along as context — here's a draft, improve it. It seemed obviously more efficient.
It anchors, badly. The strong model inherits the cheap one's framing, keeps its structure, and corrects wording rather than reasoning. On the escalations we hand-checked, the "improve this draft" path was worse than a clean run about a third of the time — and it was worse in the specific way that matters, because it repeated the mistake that triggered the escalation while polishing the prose around it.
Escalation is not a retry. It's a fresh attempt by someone better. Send the original request.
Per 1,000 requests, at our mix:
| All-strong | Cheap-first + gate | |
|---|---|---|
| Requests attempted on cheap | 0 | 950 |
| Escalated | — | 143 |
| Requests touching the strong model | 1,000 | 193 |
| Cost | $14.50 | $4.16 |
71% off. 81% of requests are served entirely by a model that costs a tenth as much, and the 5% we force to the strong model never enter the router at all.
The forced list is short and it is a policy decision, not a measurement: anything a user is going to send to a customer, anything that writes to production, and anything in the app builder's codegen path. Those never touch the cheap model regardless of what a gate would have said.
Result.Model
from day one if there's any chance you're in this bucket — retrofitting it is miserable.You don't need the whole thing to get most of the money.
Result.Model
in your logs and a kill switch on the config.Then measure the escalation rate per request kind, because that's the number the whole design lives on — and it's the number that tells you which job to move next.
The pitch for cheap models in 2026 isn't that they got good enough to replace the frontier. It's that they got cheap enough that checking whether they were good enough is now free.