{"slug": "i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s", "title": "I put my cost router on a neutral benchmark. It ranked near the bottom, and that's the interesting part", "summary": "A developer repriced three months of Claude Code usage—39.5 billion tokens across 139,835 requests—and found that routing to cheaper models could cut costs by 50-58%. They built OmnisRouter, a cost router that sends each request to the cheapest capable model, and tested it on RouterArena, an independent benchmark. Despite ranking near the bottom on cost, the developer argues the benchmark's cost axis favors cheap open-model pools, not routing skill, and that premium-model routers like theirs start at a disadvantage.", "body_md": "A few weeks back I repriced three months of my own Claude Code usage. Real traffic, not a demo: 39.5 billion tokens across 139,835 requests. At API rates that's about $30,000, and 91% of it went to Opus because that's what the default reaches for. Route the fraction a smaller model handles as well, and the bill drops by roughly 50 to 58%. Five or six grand a month, on work that was already done and graded.\n\nThat's why OmnisRouter exists. It sits in front of your agents, reads each request, and sends it to the cheapest model that can answer it. The full write-up and the method are [in the flagship post](https://dev.to/fortitudeomnis/i-repriced-40-billion-tokens-of-real-ai-coding-the-bill-goes-where-nobody-tells-you-453h).\n\nSaving your own money on your own data is one thing. I wanted an outside check on somebody else's turf, with somebody else's scoring. So I took the router to RouterArena.\n\nRouterArena is an independent benchmark for LLM routers, out of an ICLR 2026 paper. It's a proper piece of work: 809 queries in the quick split, spread across 39 datasets, everything from MMLU-Pro and code generation to translation, chess and medical multiple-choice. You plug your router in, it picks a model per query, the harness runs that model and grades the answer, and you get an accuracy number, a cost number, and a combined Arena score. There's a hard rule I stuck to: you evaluate on their data, you never tune on it. No gaming the test.\n\nI wired OmnisRouter in with the pool I care about: GPT-5, Opus-5, Claude Haiku, and GPT-5-nano. The models a team running coding agents genuinely uses.\n\nThe first scored run came back at 42% accuracy, which was obviously wrong. So before I say a word about the result, here's what I found when I opened it up.\n\nOpus was scoring 0% on every single query. The scoring path was grabbing the model's extended-thinking block instead of its answer text, so the grader was marking a raw thinking object wrong 40 times out of 40. Second, my query file was missing the per-dataset answer-format instruction the grader keys on, so most models answered correctly in prose and got marked wrong because the grader couldn't find the answer in the shape it expected.\n\nBoth fixed. Opus went from 0% to 85%, the overall number went from 42% to 72.7%, and abnormal entries went from 558 to zero. If you run a benchmark and the number looks too bad to be true, it usually is, and it's usually your plumbing.\n\nOn its own pool, OmnisRouter scores 72.7% accuracy at $3.71 per thousand queries, for an Arena score of 0.669.\n\nThat ranks it 16th out of 18. Near the bottom. On the easier split, even.\n\nI could stop there and let you think I'm burying it, so I won't. Here's the top of the cost board next to us:\n\n```\nrouter              accuracy   cost/1K   Arena\nPaix2-router          79.7%    $0.27     0.776\ncross-router          78.2%    $0.29     0.762\nsqwish-router         76.4%    $0.16     0.754\nhybrid-router         71.4%    $0.04     0.721\nomnisrouter           72.7%    $3.71     0.669\n```\n\nLook at the cost column. The leaders come in at four cents to twenty-nine cents per thousand. We come in at $3.71. That's the whole gap, and it isn't routing skill.\n\nI went and looked at what the top routers route to. Every one of them runs a cheap open-model pool:\n\nNot one of the leaders uses GPT-5, Claude, or any frontier model. And that makes sense, because the Arena cost axis rewards spend, so the way to win it is to route to the cheapest models that still clear the accuracy bar on an academic test set. Fair enough as a benchmark design. But it means the board measures how cheap your pool is at least as much as how well you route it, and a router built for the premium models people run their agents on starts the race carrying a piano.\n\nI'm not knocking the benchmark, to be clear. It's careful and the accuracy scoring is sound. I'm saying the cost ranking answers a different question than the one I care about.\n\n\"The board's unfair to premium pools\" is a convenient thing for me to say, so I ran the check that could prove me wrong. RouterArena also computes optimality metrics: how close each router gets to the oracle, the best model per query, within its own pool. That strips out the cheap-pool advantage entirely and measures routing quality on its own.\n\nHere's where we land:\n\n```\nrouter            Opt.Sel   Opt.Acc\nPaix2-router       89.7%     100.0%\nBARouter           64.4%      93.8%\nagentforge         51.1%      98.7%\nauto_router        37.7%      86.0%\nr2-router          24.5%      99.8%\nomnisrouter        23.7%      81.5%\n```\n\nOpt.Sel is how often the router picks the cheapest correct model. Opt.Acc is how much of its pool's achievable accuracy it captures. We're 8th of 13 on selection, and last of 13 on accuracy captured.\n\nThat last number is the honest one. 81.5% means OmnisRouter leaves more winnable accuracy on the table than anyone else on the board, and it does that on purpose. It's a cost-first router. Faced with a query a cheaper model can probably handle, it takes the cheaper model and the saving, even when the strongest model in the pool would have nudged the answer over the line. Tune it to chase accuracy and that number climbs, but then it stops being the thing I built. Some of the high selection scores above come from routers that barely route at all, by the way: park 95% of traffic on one cheap model that's usually the cheapest-correct and you \"win\" selection by barely choosing at all. I'd rather show you my mediocre 23.7% and tell you why than dress that up as skill.\n\nA leaderboard rank isn't the point, and I'm not going to pretend a benchmark built around cheap open models is the scoreboard OmnisRouter should be judged on. The point is the bill.\n\nOn real Claude Code traffic, on the frontier models a working team runs every day, routing the cheaper-capable fraction cuts the cost by about half. The bill and the 91% Opus share are measured from three months of real requests. The saving is what comes off that bill once you route the work a smaller model handles, sized from the fraction OmnisBench measures as routable, so it's a reprice rather than a reroute I ran end to end. No academic test set moves it, and no cost board can rank it, because none of them run the models the work is happening on.\n\nIf your agent bill has a comma in it and you don't know where the money goes, that's the problem worth solving. The leaderboard was me checking my own work in public. Turns out the honest version is more useful than a good rank would've been.\n\nOmnisRouter and the OmnisBench benchmark behind it are both open, Apache-2.0, and the numbers above are reproducible with RouterArena's own harness. Bring your own keys and check me.", "url": "https://wpnews.pro/news/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s", "canonical_source": "https://dev.to/fortitudeomnis/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-thats-the-interesting-6bc", "published_at": "2026-08-26 13:37:36+00:00", "updated_at": "2026-08-26 13:45:10.708867+00:00", "lang": "en", "topics": ["ai-infrastructure", "ai-tools", "large-language-models", "developer-tools"], "entities": ["OmnisRouter", "RouterArena", "Claude Code", "GPT-5", "Opus-5", "Claude Haiku", "GPT-5-nano"], "alternates": {"html": "https://wpnews.pro/news/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s", "markdown": "https://wpnews.pro/news/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s.md", "text": "https://wpnews.pro/news/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s.txt", "jsonld": "https://wpnews.pro/news/i-put-my-cost-router-on-a-neutral-benchmark-it-ranked-near-the-bottom-and-that-s.jsonld"}}