Startups are quietly cutting their LLM bills in half by routing prompts to cheaper models instead of sending every request to the most expensive one.
Most founders paying an LLM bill are overpaying for a simple reason: they're sending every prompt, from "summarize this email" to "debug this distributed systems race condition," to the same expensive model. How does AI model routing work? It sits a lightweight classifier or scoring layer in front of your LLM calls, reads each prompt before it goes anywhere, and sends it to the cheapest model that can actually handle it. Simple questions go to a small, fast model. Hard ones go to the expensive one. You stop paying frontier prices for questions a $0.15-per-million-token model could have answered just as well.
This isn't a hypothetical efficiency trick. It's already built into the products you use. When OpenAI shipped GPT-5 in August 2025, it didn't ship one model, it shipped a router. Sam Altman described a "real-time router" that decides, prompt by prompt, whether to hand your question to a fast model or kick it up to a deeper reasoning model, based on conversation type, complexity, and whether you explicitly ask it to "think hard." You never see the switch happen. You just get an answer, and OpenAI gets to run a huge share of its traffic through the cheap path.
- Model routing sends each prompt to the cheapest model capable of answering it correctly, instead of a single model handling every request
- OpenAI built a router directly into GPT-5 in August 2025 that silently switches between fast and reasoning models based on prompt difficulty
- Startups like Martian and Not Diamond sell routing as a standalone layer, claiming 20 to 98 percent cost cuts by classifying prompts before they hit any model
- Most routers work by scoring prompt complexity with a small classifier model, then picking from a ranked list of LLMs based on cost, latency, and accuracy thresholds
- Routing fails silently when a hard prompt gets misclassified as easy, so teams that skip evaluation logging often don't notice quality loss until a customer complains
Strip away the marketing and a model router does three things, in order. First, it scores the incoming prompt for difficulty, using signals like token length, the presence of code, whether the question requires multi-step reasoning, or a similarity match against prompts it's seen before. Second, it checks a ranked list of candidate models, usually three to six of them, each tagged with a cost per token, an average latency, and a track record of accuracy on that kind of task. Third, it picks one and sends the request. The whole decision happens in milliseconds, before your user notices anything.
The scoring step is the part people get wrong when they imagine this system. It isn't the expensive model grading the prompt first, because that would defeat the purpose. It's usually a much smaller, purpose-built classifier, sometimes a fine-tuned BERT-sized model, sometimes a set of hand-written rules, sometimes a tiny LLM call that costs a fraction of a cent. Not Diamond, a routing startup that came out of stealth in 2024, trains its router on labeled examples of which model performed best on which prompt type, then uses that training data to predict, for a brand new prompt, which model is likely to give the best answer for the lowest price. Martian, another routing company, takes a related approach: it builds what it calls a "model mapping" that predicts how GPT-4, Claude, and open-weight models will each perform on a given request before any of them actually run it.
How Does AI Agent Memory Work Across Sessions, and Why It Keeps Failing How does AI agent memory work across sessions? It doesn't, not really: what looks like memory is a context window, a vector database, and a summarizer passing notes to each other between calls. This guide breaks down each piece, where it fails, and how companies like Letta, Mem0, and OpenAI are patching around the gaps. - how AI agents remember information between conversations - why AI agent memory systems fail in production
Once the router has an answer, it doesn't stay static. Most production routers log every decision along with the eventual output quality, whether measured by a user thumbs-up, a downstream test passing, or a separate evaluation model grading the answer. That feedback loop is what separates a real router from a crude if-statement. A team that hardcodes "route anything under 50 words to GPT-4o-mini" isn't running a router. They're running a guess.
The price gap between models is large enough that routing correctly changes a company's unit economics. As of 2025, Anthropic charges $15 per million output tokens for Claude Opus and roughly $0.80 for Claude Haiku, a nearly 19x spread for two models built by the same lab. OpenAI's GPT-4o costs multiples of GPT-4o-mini. If even half of a startup's traffic is genuinely simple, classification tasks, short summaries, formatting requests, FAQ answers, routing that half to the cheap tier can cut a monthly inference bill close to in half without touching the hard half at all.
Frankly, most AI products don't need Opus-level reasoning for most of what they ask it to do. A support bot that mostly answers "what's your refund policy" and occasionally has to reason through a multi-order billing dispute is paying frontier prices for the ninety percent of traffic that never needed them. Vendors selling routing as a service lean hard on this gap. Martian has publicly claimed cost reductions of up to 98% on API spend for some customers by routing away from the most expensive models whenever a cheaper one would produce an equivalent answer, and OpenRouter, which aggregates access to dozens of models through one API, built its own automatic routing feature specifically so developers stop hand-picking a model per request.
None of that means routing is free money. It costs something to run the classifier on every request, and that overhead, plus the added latency of a routing decision, has to be smaller than the savings it produces or you've just added a layer of complexity for nothing.
There's a latency tax too, and it's the part vendors mention last. A round trip to a classifier before the real model call adds anywhere from a few milliseconds, for a rules-based check, to a few hundred, for a small model doing the scoring. For a chat product where users are already waiting on a stream to start, that's tolerable. For a voice agent or anything latency-sensitive, it can eat the entire benefit, since a customer on a phone call notices a half-second stall a lot faster than they notice a line item on your AWS bill.
Where founders actually build this, without buying a platform #
You don't need to sign up for a routing vendor to get the basic version working. The simplest form of model routing, and the one most early-stage teams actually ship first, is a hardcoded tier system: classify by task type rather than by scoring the prompt itself. A coding agent might route every "explain this error" request to a cheap model and every "refactor this 400-line file" request to a frontier one, based purely on which tool call the user invoked. Cursor and other AI coding tools quietly do versions of this, using smaller, faster models for autocomplete-style suggestions and reserving the larger models for full-file edits or chat-based reasoning, because running a frontier model on every keystroke would be both too slow and too expensive to ship as a product.
The next step up is a difficulty classifier trained on your own data rather than someone else's. If you're running a customer support product, you already have thousands of past tickets with resolution outcomes. Feed those into a small classifier, tag which model tier resolved each one correctly, and you have a training set for a router specific to your traffic, not a generic one built for someone else's prompt distribution. This is closer to what Not Diamond and Martian sell as a managed product, but it's buildable in-house with an afternoon and a spreadsheet if your volume is modest.
How Does Prompt Caching Work for LLMs, and Why It Cuts Bills in Half How does prompt caching work for LLMs? It's a token-storage system with its own write and read pricing, its own expiration clock, and a hidden minimum size that fails silently if you miss it. Get the prompt order wrong and you pay full price every time, even with caching turned on. - how prompt caching works for language models - why prompt caching reduces LLM API costs significantly
Portkey and OpenRouter sit somewhere between those two options. Both act as a single API gateway in front of dozens of models, and both let you set routing rules, fall back to a second model if the first one errors or times out, and cap spend per model per day. That's a meaningfully lower lift than training your own classifier, and for a team of two or three engineers trying to ship a product instead of an ML infrastructure layer, it's usually the right place to start before building anything custom at all.
The failure mode nobody advertises #
Routing breaks quietly. A misclassified prompt doesn't throw an error, it just gets answered badly by a model that was never going to be good enough, and unless you're logging outcomes, nobody notices until a customer does. This is the actual risk in cheap vs expensive LLM routing setups: the failure isn't a crash, it's a slow erosion of answer quality that shows up in churn numbers three months later, long after anyone remembers to check the routing logs.
The fix is not complicated, but it's the part teams skip because it isn't the interesting half of the build. Every routed request needs a quality signal attached to it, even a cheap one like an automated eval model scoring the output on a 1 to 5 scale, so you can measure whether the cheap tier is actually holding up on the traffic it's been handed. Teams that skip this step tend to discover the problem the hard way: a support bot that saved 40% on inference costs but started giving wrong refund amounts on the 15% of tickets that were more complex than the router assumed.
There's also a harder version of this problem for anyone building a model router for AI agents rather than a single-turn chatbot. An agent chains multiple LLM calls together, plan, execute, verify, and a routing mistake early in that chain compounds. Send the planning step to a model too weak to break the task down correctly, and every downstream step inherits a bad plan, even if each individual step gets routed to a perfectly capable model. This is why the more sophisticated routing setups score difficulty per step in a chain, not once per session, and why a router built for single-shot Q&A often falls apart the moment you point it at an agent instead of a chatbot.
What this actually changes for a founder shipping today #
If you're building an AI product right now, the practical takeaway isn't "buy a router." It's that treating every request as equally hard is the default mistake, and it's an expensive one to leave uncorrected. Start with the crude version: tag your product's most common request types, figure out which ones a $0.15-per-million-token model handles just as well as a $15-per-million-token one, and hardcode that split before you build anything fancier. That alone captures most of the savings routing vendors advertise, without adding a new vendor dependency or a new point of failure. Only reach for a learned classifier, whether you build it or buy it from Martian, Not Diamond, or a similar vendor, once you have enough volume and enough logged outcomes to actually train one well. A classifier trained on a hundred examples isn't smarter than a hardcoded rule, it's just a hardcoded rule wearing a machine learning costume. The savings from model routing are real and the mechanism is genuinely simple once you see it laid out, but the discipline of measuring what the cheap model actually gets wrong is the part that makes the whole thing worth doing instead of just cheaper and quietly worse.
Also read: How Reg CF Equity Crowdfunding Actually Works for Startups Under $5 Million • How the R&D Tax Credit Offsets Payroll Taxes for Startups With No Revenue • How Does AI Agent Memory Work Across Sessions, and Why It Keeps Failing
This article is posted in AI News, check it out for more related stories.
Join the discussion #
Open in the community → Almost there. Sign in and your reply posts straight away.