{"slug": "40-lines-of-go-that-cut-our-llm-bill-by-71", "title": "40 Lines of Go That Cut Our LLM Bill by 71%", "summary": "A developer at an unnamed company built a 40-line Go router that cut their LLM bill by 71% by sending most requests to a cheaper model and using output-based gates to escalate only when needed. The system runs the cheap model first and escalates to a stronger model only if the output fails quality gates, leveraging the fact that escalation rates below 90% make the approach cost-effective.", "body_md": "On July 30, OpenAI cut GPT-5.6 Luna to **$0.20 per million input tokens and $1.20 per million output** — down from $1 and $6. An 80% cut. Azure matched it on August 1.\n\nWe did what most teams did with that news: nothing. Our gateway sent every request to the strong model, because it always had, and because \"just use the cheap model\" sounds like a decision that comes back as a support ticket three weeks later.\n\nThen someone put the bill next to the traffic mix, and the awkward part was obvious. The overwhelming majority of our requests were **title this thread**, **summarise this diff**, **extract the fields from this form**, **name this file**. We were paying frontier prices to generate three-word document titles.\n\nHere's what we shipped instead. It's about forty lines of Go, it moved 81% of requests off the expensive model, and it cut the bill by 71%.\n\nIt also broke four things, and those are the interesting part.\n\nThe obvious design — the one everyone writes first — is a classifier in front of the router:\n\nLook at the incoming request. Decide if it's\n\neasyorhard. Send it to the model that matches.\n\nWe built this. It's worse than it looks, for three reasons.\n\n**You need a model to run the classifier.** Now every request pays an extra call before any work happens. A cheap classifier is another Luna call and another 300ms; an accurate classifier is a strong-model call, which is the cost you were trying to avoid.\n\n**Short is not easy.** \"Fix the timezone bug\" is eleven characters and needs everything you've got. \"Summarise the following 4,000-word RFC\" is long and trivial. Length, token count, keyword lists — every cheap heuristic we tried correlated with the *shape* of the request and not with the difficulty of it.\n\n**You're predicting the answer before you've seen it.** That's the actual problem. Difficulty is a property of the work, and the only honest way to learn it is to do the work.\n\nThe rule:don't classify the prompt. Run the cheap model and judge the output.\n\nThe design that works is embarrassingly simple:\n\nThe thing that makes this viable is arithmetic that surprised us. At our average request shape — roughly 3,000 input tokens and 700 output — a Luna call costs about **$0.0014** and a strong-model call about **$0.0145**. Ten to one.\n\nSo a request that gets escalated costs `0.0014 + 0.0145 = $0.0159`\n\ninstead of `$0.0145`\n\n. About 10% more.\n\nWork out the break-even escalation rate:\n\n```\ncheap + (p × strong) < strong\np < (strong − cheap) / strong\np < (0.0145 − 0.0014) / 0.0145\np < 0.90\n```\n\n**You would have to escalate nine times out of ten before the cheap attempt costs you money.** We escalate 15% of the time. Cost is not the constraint here, and if you're arguing about whether cheap-first is worth the double billing, you're arguing about the wrong resource.\n\nLatency is the constraint. We'll come back to that.\n\n```\ntype Result struct {\n    Text   string\n    Model  string\n    Tokens Usage\n}\n\n// A gate returns the reason this answer can't be trusted, or \"\" to accept it.\ntype Gate func(req Request, out string, fin FinishReason) string\n\nfunc (r *Router) Do(ctx context.Context, req Request) (Result, error) {\n    if req.ForceStrong || r.alwaysStrong[req.Kind] {\n        return r.call(ctx, r.strong, req)\n    }\n\n    cheap, err := r.call(ctx, r.cheap, req)\n    if err != nil {\n        // Availability, not quality. Fall through, don't fail the request.\n        return r.call(ctx, r.strong, req)\n    }\n\n    for _, gate := range r.gates {\n        reason := gate(req, cheap.Text, cheap.Finish)\n        if reason == \"\" {\n            continue\n        }\n        r.metrics.Escalate(req.Kind, reason)\n\n        strong, err := r.call(ctx, r.strong, req) // NB: req, not req+cheap.Text\n        if err != nil {\n            return cheap, nil // degraded, not failed\n        }\n        return strong, nil\n    }\n\n    r.metrics.Accept(req.Kind)\n    return cheap, nil\n}\n```\n\nTwo lines in there are load-bearing and neither is obvious.\n\n`return r.call(ctx, r.strong, req)`\n\nwhen the *cheap call errors* — a 429 or a timeout is an availability problem, and availability problems should not surface to the user as a failed request when you have a second provider sitting right there.\n\n`return cheap, nil`\n\nwhen the *strong call errors* — you already have an answer. It may be a worse answer. Shipping a worse answer beats shipping a spinner.\n\nAnd the comment on the escalation call is the one people get wrong, which is failure mode #4 below.\n\nThe gates are the whole product. The router is plumbing.\n\n``` js\nvar defaultGates = []Gate{\n    SchemaInvalid,   // had to be JSON matching a schema, and wasn't\n    ToolArgsMissing, // called a tool, omitted a required argument\n    Truncated,       // finish reason != stop\n    EmptyOrHedged,   // under 24 chars, or matches the hedge set\n}\n```\n\nThey are ordered by how cheap they are to evaluate, and every one of them is **structural**. None of them asks a model to grade another model.\n\n** SchemaInvalid** does the most work by a distance. Anything with a defined output shape — field extraction, classification, structured summaries — gets validated against the schema you already have. If it doesn't parse or doesn't conform, escalate. This gate alone catches about 60% of our escalations.\n\n** ToolArgsMissing** is the same idea for function calls. The cheap model picks the right tool far more reliably than it fills in the right arguments, and a missing required argument is a free, exact signal.\n\n** Truncated** is one field comparison and people skip it constantly. A\n\n`finish_reason`\n\nof `length`\n\nmeans you have a sentence that stops mid-** EmptyOrHedged** is the weakest one, and I want to be specific about how weak, because it's the one everybody wants to build first.\n\nWe started with the intuitive version: ask the cheap model to say when it isn't confident, then escalate on that. It fired on **0.4%** of responses. Our measured error rate on the same traffic was around 15%. The model's self-reported uncertainty was not a signal, it was decoration.\n\nWhat actually works in that slot is a small, boring list: empty, under 24 characters, or an exact-ish match against a hedge set you build by reading two hundred real failures (`\"I don't have enough information\"`\n\n, `\"As an AI\"`\n\n, `\"Could you clarify\"`\n\n). Not confidence. Refusal.\n\nThe rule:gate on structure you can check, not on the model's opinion of itself.\n\nThis was immediate and it's the real cost of the design.\n\n| All-strong | Cheap-first + gate | |\n|---|---|---|\n| p50 | 2.9s | 1.4s |\n| p95 | 7.8s | 9.6s |\n| p99 | 11.2s | 16.4s |\n\nLuna is fast, so the 85% that get accepted got much faster. The 15% that escalate pay for both calls, serially, and they land in your tail.\n\nIf you have a latency SLO, that tail is where the design either survives or doesn't. Two things helped: run the gates on the *streamed head* rather than the finished response where you can, and put a hard `escalateBudget`\n\non the clock — if the cheap call already burned 4 seconds, return it and log the miss rather than starting a second call you can't afford.\n\nThe whole architecture assumes you get to look at the output before deciding. Streaming assumes you've already committed.\n\nThere is no clever fix, only a choice:\n\nMost of the money was in the first bucket anyway. Structured background work is high volume and nobody is watching a cursor blink at it.\n\nThe output is fluent. It's well-formatted. It uses your headings, it hits your tone, it's the right length. It's just wrong.\n\nThat's why \"does this look like a good answer\" gates — including LLM-as-judge in the hot path — did badly for us. Fluency is exactly the axis where the price gap has closed most. Judgement, multi-step reasoning, and *knowing what it doesn't know* are where it hasn't closed at all.\n\nStructural gates work because they don't have an opinion. Valid JSON is valid JSON.\n\nOur first version passed the failed attempt along as context — *here's a draft, improve it*. It seemed obviously more efficient.\n\nIt anchors, badly. The strong model inherits the cheap one's framing, keeps its structure, and corrects wording rather than reasoning. On the escalations we hand-checked, the \"improve this draft\" path was worse than a clean run about a third of the time — and it was worse in the specific way that matters, because it repeated the mistake that triggered the escalation while polishing the prose around it.\n\nEscalation is not a retry. It's a fresh attempt by someone better. Send the original request.\n\nPer 1,000 requests, at our mix:\n\n| All-strong | Cheap-first + gate | |\n|---|---|---|\n| Requests attempted on cheap | 0 | 950 |\n| Escalated | — | 143 |\n| Requests touching the strong model | 1,000 | 193 |\n| Cost | $14.50 | $4.16 |\n\n**71% off.** 81% of requests are served entirely by a model that costs a tenth as much, and the 5% we force to the strong model never enter the router at all.\n\nThe forced list is short and it is a policy decision, not a measurement: anything a user is going to send to a customer, anything that writes to production, and anything in the app builder's codegen path. Those never touch the cheap model regardless of what a gate would have said.\n\n`Result.Model`\n\nfrom day one if there's any chance you're in this bucket — retrofitting it is miserable.You don't need the whole thing to get most of the money.\n\n`Result.Model`\n\nin your logs and a kill switch on the config.Then measure the escalation rate per request kind, because that's the number the whole design lives on — and it's the number that tells you which job to move next.\n\nThe pitch for cheap models in 2026 isn't that they got good enough to replace the frontier. It's that they got cheap enough that *checking whether they were good enough* is now free.", "url": "https://wpnews.pro/news/40-lines-of-go-that-cut-our-llm-bill-by-71", "canonical_source": "https://dev.to/infoinlet1/40-lines-of-go-that-cut-our-llm-bill-by-71-4do1", "published_at": "2026-08-30 06:49:55+00:00", "updated_at": "2026-08-30 07:23:32.217074+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops"], "entities": ["OpenAI", "Azure", "Luna"], "alternates": {"html": "https://wpnews.pro/news/40-lines-of-go-that-cut-our-llm-bill-by-71", "markdown": "https://wpnews.pro/news/40-lines-of-go-that-cut-our-llm-bill-by-71.md", "text": "https://wpnews.pro/news/40-lines-of-go-that-cut-our-llm-bill-by-71.txt", "jsonld": "https://wpnews.pro/news/40-lines-of-go-that-cut-our-llm-bill-by-71.jsonld"}}