{"slug": "everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers", "title": "Everyone is saving 60% on LLM costs. Nobody will show you the numbers.", "summary": "The developers behind the open-source OmnisBench project have released a benchmark designed to measure the true cost savings of LLM routing. Running 364 coding and math tasks across four models, they found that ideal routing achieved 99.7% task success at $0.62 per 1,000 requests, a 90% cost reduction compared to always using the frontier model. However, they caution that the cheapest model alone already scored 94.5%, so routing's real benefit is the last few points of quality, not a magical 60% saving.", "body_md": "Every few weeks the same post does the rounds. Someone's routing layer cut their LLM bill by 60%, \"with no drop in quality\", and here's a lovely chart to prove it.\n\nExcept it doesn't prove anything. The eval set is private. The grading is private. The actual model responses that produced that score are never published. You're being asked to trust a screenshot, and we've been in I.T. long enough to know roughly what a screenshot is worth.\n\nSo we built the thing that was missing. It's called OmnisBench, it's open source (Apache 2.0), and its whole job is to measure how good LLM routing actually is, in a way you can check for yourself.\n\nIf you're not knee deep in this yet: an LLM router looks at each request and sends it to the cheapest model that can still do the job, rather than firing everything at one expensive frontier model. Easy prompts go to something small. Hard ones go to the big model. In theory you keep most of the quality and pay a fraction of the bill.\n\nThat's the theory. OmnisBench measures the practice.\n\nThe number we care about most is what we call the oracle. For every task, the oracle picks the cheapest model that actually got it right. We know which ones did, because we ran them all. That is not a router you can ship. It's the ceiling. It's the best any router could possibly do on a given set of tasks, and it tells you how much money is genuinely on the table before anyone starts making claims.\n\nWe ran it over 364 tasks: HumanEval (164 coding problems, graded by their own unit tests) and GSM8K (200 maths problems, graded on the answer). The model pool was Claude Opus 5, GPT-5, Claude Haiku 4.5 and GPT-5-nano.\n\nHere is what came out.\n\n| Policy | What it does | Task success | Cost per 1,000 requests |\n|---|---|---|---|\n| oracle | cheapest model that actually solved each task | 99.7% | $0.62 |\n| always-frontier | everything to Claude Opus 5 | 99.2% | $6.25 |\n| random | pick a model at random | 96.2% | $4.01 |\n| always-cheapest | everything to GPT-5-nano | 94.5% | $0.43 |\n\nSo the headline writes itself. Ideal routing hit 99.7% success for roughly 90% less than sending everything to the top model. Very nice.\n\nNow let me talk you out of it a little.\n\nTwo things you will not see on most of those 60% charts.\n\nFirst, the oracle is a ceiling, not a product. It cheats. It knows, after the fact, which model got each task right, and no live router has that luxury. The real score for a router you can actually deploy is how close it gets to that oracle line, and the gap is the interesting part. Anyone quoting an oracle-style number as their shipping product is selling you the ceiling.\n\nSecond, and this is the one that surprised us: the cheapest model on its own already scored 94.5%. On this particular set of tasks GPT-5-nano is just quietly good. So routing is not buying you a magic 60%. It's buying you the last few points of quality without paying frontier prices for all of it. That is still very much worth having. It just isn't the fairy tale.\n\nWe would rather tell you that up front than have you find out after you have quoted the shiny number to your boss.\n\nThis is the whole reason OmnisBench exists. It does not ask you to trust the table above. It publishes the actual model response for every task, and ships a command that re-grades the lot offline.\n\n```\npip install -e .\npython scripts/prepare_datasets.py                                  # HumanEval + GSM8K\npython -m omnisbench.cli run    --config configs/v0.yaml --run runs/mine\npython -m omnisbench.cli report --run runs/mine\npython -m omnisbench.cli verify --run runs/2026-08-19               # re-grades the published run, no API calls\n```\n\n`verify`\n\nre-runs the graders against the published responses and rebuilds the entire leaderboard, cost included, without calling a single API or needing a key. Tamper with one stored answer and it fails. That is the point. If our numbers are wrong, you can prove it in about a minute, and we would genuinely like you to try.\n\nThe results, the method and the honest caveats are all on the site, and the code is on GitHub:\n\nWe are also building an actual open router, OmnisRouter, which this benchmark will grade on every release, so it can't quietly regress and hope nobody notices. More on that when it's ready.\n\nIf you can break the numbers, poke a hole in the method, or you just reckon we've measured the wrong thing, tell us. That is what open and verifiable is for.\n\nOmnisBench is built by [Fortitude Omnis](https://fortitude-omnis.group). We make small, sharp tools, and we try not to lie in our own marketing.", "url": "https://wpnews.pro/news/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers", "canonical_source": "https://dev.to/fortitudeomnis/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers-5e7j", "published_at": "2026-08-19 13:05:03+00:00", "updated_at": "2026-08-19 13:12:50.643573+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-research", "ai-tools", "mlops"], "entities": ["OmnisBench", "HumanEval", "GSM8K", "Claude Opus 5", "GPT-5", "Claude Haiku 4.5", "GPT-5-nano"], "alternates": {"html": "https://wpnews.pro/news/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers", "markdown": "https://wpnews.pro/news/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers.md", "text": "https://wpnews.pro/news/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers.txt", "jsonld": "https://wpnews.pro/news/everyone-is-saving-60-on-llm-costs-nobody-will-show-you-the-numbers.jsonld"}}