{"slug": "the-llm-router-was-a-2024-idea", "title": "The LLM Router Was a 2024 Idea", "summary": "Manifest, an open-source development platform, shipped an LLM router in March 2024, saw 7,000 cloud users use it for four months, and deprecated it in June, with full shutdown on September 1. The company's post-mortem attributes the failure to structural problems: agentic workloads make per-request routing ineffective because the prompt alone doesn't reveal task complexity, and prompt caching (priced at 10% of base input cost on Anthropic) makes switching models mid-session more expensive than staying put. The analysis suggests that the 2024 arbitrage of routing between cheap and expensive models has eroded as the price-quality gap narrowed and workloads shifted from single-shot chat to multi-step agent sessions.", "body_md": "[AI](https://sourcefeed.dev/c/ai)Article\n\n# The LLM Router Was a 2024 Idea\n\nManifest killed its router after four months, and its post-mortem indicts the whole category.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n[Manifest](https://manifest.build/) shipped an LLM router in March, watched 7,000 cloud users bounce off it for four months, and deprecated it in June. It shuts down for good on September 1. That's a fast funeral for a feature category that half the AI infrastructure world is still pitching — and the reasons Manifest gives aren't quirks of their implementation. They're structural problems with the whole idea of per-request cost routing, and they've been getting worse, not better, for a year.\n\nThe candor is worth rewarding with scrutiny, so let's scrutinize: they're mostly right, and the interesting part is *why* an idea that genuinely worked in 2024 stopped working.\n\n## The arbitrage was real — in 2024\n\nThe router pitch came from a real place. When LMSYS published [RouteLLM](https://lmsys.org/blog/2024-07-01-routellm/) in mid-2024, the frontier looked like GPT-4 at premium prices with a huge quality gap down to Mixtral 8x7B at a fraction of the cost. Train a classifier on Chatbot Arena preference data, send the easy 74% of queries to the cheap model, and the benchmarks obliged: 85% cost reduction on MT Bench while keeping 95% of GPT-4's quality. That's a genuine arbitrage, and a wave of startups and gateway features got funded on it.\n\nTwo things about that setup mattered more than anyone admitted. The workload was single-shot chat — one prompt in, one answer out, complexity fully visible upfront. And the price-quality spread between tiers was enormous, so even a sloppy classifier printed money.\n\nNeither condition holds anymore.\n\n## Agents broke the input\n\nManifest's router classified each request into four complexity tiers — simple, standard, complex, reasoning — and routed across Anthropic, OpenAI, DeepSeek, and Mistral. Their core finding: the prompt alone doesn't contain the task. It's just the trigger. In an agentic loop, the actual difficulty emerges three tool calls in, when the \"rename this variable\" request turns out to touch a metaprogramming layer, or the \"quick summary\" requires reconciling contradictory search results.\n\nA classifier scoring the initial prompt is making its decision at the moment of maximum ignorance. The Hacker News thread on the post landed on the same point independently: \"evaluate the tests\" means nothing until you know whether the repo is a personal HTML5 site or the Linux kernel — and the router can't know, because that context arrives later, through tools.\n\nThis is the quiet way agents have rearranged the stack. Plenty of 2024-era infrastructure assumed the request was the unit of work. Agents made the *session* the unit of work, and per-request decisions — routing, but also rate limiting and cost attribution — degrade accordingly.\n\n## Caching inverted the economics\n\nThe second problem is nastier because it's arithmetic. [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) now prices cache reads at roughly 10% of base input cost on Anthropic, with OpenAI offering comparable discounts on its recent models. Agentic workloads are cache heaven: a long, stable prefix of system prompt plus tool definitions plus accumulated context, replayed on every turn.\n\nSwitch models mid-session and that entire cached prefix evaporates. You pay full freight to rebuild context on the new provider — often more than the routing decision saved. Manifest found their cost-aware router converging on the correct strategy: don't switch. A router whose optimal policy is to stop routing is a deprecation notice that writes itself.\n\nRun the numbers on a typical agent turn — say 50k tokens of accumulated context, 500 tokens of new input. Cached, the context costs you 10% of list price. A \"smart\" switch to a model that's 40% cheaper per token still loses badly, because it pays 100% on all 50k. The cheaper the caching gets, the dumber cross-provider routing becomes. Providers surely understand that caching is a switching cost; it works on routers even better than it works on you.\n\n## Even OpenAI couldn't sell it\n\nThe strongest evidence against invisible routing comes from the company with the most training data and the most incentive to make it work. GPT-5 launched in August 2025 as a router-fronted system, an autoswitcher deciding when the thinking model was warranted. Launch day, the router malfunctioned — Sam Altman publicly blamed it for the model seeming \"way dumber\" — and the deeper complaint never went away: users had built intuitions for a specific model's behavior, and the router silently discarded them. OpenAI has been walking the invisibility back ever since, giving users explicit mode pickers.\n\nThat's Manifest's third argument at consumer scale. Model choice isn't an implementation detail; it's part of the interface. Engineers build a feel for a model's failure modes — where Claude hedges, where DeepSeek hallucinates citations — and per-request routing resets that learning to zero on every call. You also can't write meaningful evals against a moving target: which model was that regression on?\n\n## What actually survives\n\nDon't over-rotate into \"routing is dead.\" Three narrower things are alive and healthy.\n\n**Gateways.** A single API over many providers, with failover, spend caps, and data-residency policies, is boring and valuable — it's why [OpenRouter](https://openrouter.ai/) and LiteLLM keep growing, and it's what Manifest pivoted to. Regulated shops route for compliance, not cost. That's rule-based, auditable, and none of the above objections apply.\n\n**Design-time selection.** The HN consensus matches what most production teams already do: pin models per task class. Gemini for broad-knowledge Q&A, Claude for the coding agent, a small open model for classification and extraction. You make the decision once, with evals, per task — not per request, with a classifier guessing from a prompt fragment.\n\n**In-family effort knobs.** The routing dividend didn't vanish; providers internalized it. Reasoning-effort parameters and thinking budgets give you the cheap-path/expensive-path split within one model family — same prompts, same caches, same behavioral intuitions, no classifier. That's the form of \"routing\" worth adopting in 2026, and it's a config flag, not a product.\n\nThe compressed price-quality frontier seals it. The gap RouteLLM arbitraged — frontier model versus Mixtral — has narrowed to the point where mid-tier models handle most of what routers used to escalate. Shrinking spread, growing switching costs, decision point moved to design time: the generic cost router got squeezed from all three sides at once.\n\nManifest's post-mortem reads less like a product failure than a datestamp. The router was a rational bet on the 2024 frontier. The frontier moved. If a vendor is pitching you per-request complexity routing today, the question to ask is the one Manifest answered honestly: after evals, cache misses, and debugging non-determinism, where does the savings actually go?\n\n## Sources & further reading\n\n-\n[Everyone is building LLM routers, we deprecated ours](https://manifest.build/blog/why-we-deprecated-our-llm-router/)— manifest.build -\n[HN discussion: Everyone is building LLM routers, we deprecated ours](https://news.ycombinator.com/item?id=49126630)— news.ycombinator.com -\n[RouteLLM: An Open-Source Framework for Cost-Effective LLM Routing](https://lmsys.org/blog/2024-07-01-routellm/)— lmsys.org -\n[Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching)— platform.claude.com\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/the-llm-router-was-a-2024-idea", "canonical_source": "https://sourcefeed.dev/a/the-llm-router-was-a-2024-idea", "published_at": "2026-07-31 21:08:21+00:00", "updated_at": "2026-07-31 21:08:50.427085+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-agents", "large-language-models"], "entities": ["Manifest", "Anthropic", "OpenAI", "DeepSeek", "Mistral", "LMSYS", "RouteLLM", "Mariana Souza"], "alternates": {"html": "https://wpnews.pro/news/the-llm-router-was-a-2024-idea", "markdown": "https://wpnews.pro/news/the-llm-router-was-a-2024-idea.md", "text": "https://wpnews.pro/news/the-llm-router-was-a-2024-idea.txt", "jsonld": "https://wpnews.pro/news/the-llm-router-was-a-2024-idea.jsonld"}}