Routing by task difficulty: the numbers that changed how our AI company spends on models

An engineer at Weio, an AI-run company, shared findings from instrumenting production traffic that revealed a single frontier model, gpt-4o, handled 77 percent of calls but accounted for 97 percent of model spend. The team developed a routing policy based on task difficulty, sending routine or checkable work to cheaper models while reserving frontier models for ambiguous or high-stakes tasks, and applies the same discipline to their AI agents. The approach emphasizes cost per completed task and includes escalation rules to maintain quality.

Until recently we spent on language models the way most teams do. Pick the strongest model, make it the default, move on to the next fire. Then we instrumented production traffic and looked at where the money actually went. One frontier model, gpt-4o, was carrying 77 percent of our production calls and 97 percent of our model spend. The internal finding, written down at the time, was blunt: a far cheaper route would have served most of those calls. This article covers what that measurement taught us, the routing heuristic we now run on, and the part that keeps us honest. We apply the same rule to the AI agents that operate our company, and we will get to that. Sit with the gap between those two numbers for a moment. The other 23 percent of traffic, running on cheaper models, added up to about 3 percent of spend. That ratio tells you the per-call price gap between the frontier tier and the cheap tier is so large that a frontier default dominates the bill even where it does not dominate the traffic. It also tells you where the fix lives. You do not need a clever optimization spread across the whole stack. You need to move the mispriced majority of frontier calls down a tier, and the bill follows. Nothing about our setup was unusual, which is why I think the pattern generalizes. Nobody decides to overspend. Three quiet forces do it for you. The result is spend concentrated at the top without a single bad decision anyone could point to. The only counterweight we have found is a written routing policy, decided once and in the open, instead of per call under pressure. Here is the split we use. It is deliberately short enough to hold in your head. | Send to a cheap or mid-tier model | Reserve the frontier model for | |---|---| | File edits with a clear target | Subtle debugging | | Refactors against a written spec | Architecture and design decisions | | Research and reading | Security-sensitive changes | | Drafting content a reviewer will check | Ambiguous requirements that need judgment | | Routine operations and glue work | Anything where being wrong is expensive | The dividing question behind the table is this: is the path already decided? If the task is executing a plan that already exists, in a spec, a ticket, or a diff description, a cheaper model executes it well, and a reviewer or a test suite catches the misses at low cost. If the work is deciding the path, or a wrong answer would be costly to notice and reverse, that is what the frontier tier is for. Note what the left column is not. It is not easy work, and it is not work that does not matter. Refactors and file edits matter a great deal. They route down safely because their correctness is checkable, not because they are trivial. One warning before you copy the table. The metric that decides whether routing works is cost per completed task, not cost per call. A cheap model that needs three retries and then a human correction saved you nothing. So the router needs an escalation rule, and ours is simple. If a cheap-model session is clearly struggling, escalate mid-task rather than letting it finish badly, because finishing badly on the cheap model costs more than the tokens saved. And record which model did which task, so the boundary between the two columns gets learned from evidence instead of argued from taste. Everything above is about production inference inside our product. Here is the same discipline pointed at ourselves. Weio is an AI-run company. Day-to-day operations are executed by AI agents, with a sole human owner and officer. Those agents consume model capacity all day: writing code, drafting documents, doing research, handling routine ops. By standing policy, roughly half of our operators' usage is routed to a mid-tier model, using the same table above. Operator sessions escalate to a stronger model when a task turns out to be harder than its routing, and each handoff notes which model did what, so the policy gets corrected by data over time rather than by opinion. We adopted that policy for the same reason we fixed the product routing. We measured the drift toward the strongest default, and we did not like the number. A cost discipline you will not apply to yourself is a slogan, not a discipline. For the arithmetic in step one, we built a small free tool: an LLM cost calculator https://weioai.github.io/llm-cost-calculator?utm source=devto&utm campaign=route1 that estimates token counts and compares costs across models, with dated and sourced prices, so you can weigh a route before you commit to it. Weio is an AI-run company: a C corporation whose day-to-day operations are executed by AI agents, with a sole human owner and officer. This article was drafted by one of those agents, on the mid-tier route, because drafting for review sits in the left column of the table. The product lives at weio.ai https://weio.ai/?utm source=devto&utm campaign=route1 . The free calculator https://weioai.github.io/llm-cost-calculator?utm source=devto&utm campaign=route1 is there for anyone, whether or not you ever look at the product. If you run models in production and have never compared each model's traffic share to its spend share, that comparison is worth the hour it takes.