We built a benchmark, then caught it strangling the models it was grading

Fortitude Omnis Group's OmnisBench benchmark initially showed small models performing surprisingly well, but community feedback revealed potential contamination from old benchmarks like HumanEval and GSM8K. Upon testing with fresh problems from LiveCodeBench, the team discovered that their frontier model's poor scores were due to a 4,096-token output limit truncating reasoning before code generation, not model incompetence. After fixing the budget, the frontier model scored 86.7% on fresh problems, while the cheap model dropped to 60%, confirming contamination and highlighting routing's value on unseen tasks.

A couple of day ago I posted about OmnisBench, our open benchmark for LLM routing, specifically our LLM Router OmnisRouter https://omnisbench.fortitude-omnis.group/ , and made a fuss about how you can re-grade every number yourself because we publish the actual model responses. Two commenters, deanlee and jugeni, very politely pointed out that the whole thing might be resting on a fib. Their point: HumanEval and GSM8K are old. The models have almost certainly read the answers. So when I cheerfully reported that the cheapest model already scores 94.5% and routing only recovers the last few points, that's maybe less "small models are quietly brilliant" and more "small models have seen the exam paper." Fair. Annoyingly fair. So we did the thing you're supposed to do and actually built the fresh split we said we would. The idea's simple. Take LiveCodeBench, which stamps every problem with a release date, and keep only the ones published after the models could plausibly have trained on them. Grade the same routing policies on the old, probably-memorised tasks and on the new, definitely-not tasks, side by side. If the routing story only survives on the old set, you deserve to know. I ran it. Here's where it took a turn. They came back grim. The cheap model fell off a cliff. Even the big expensive frontier model, the one whose entire job is to be good at this, was landing well under where a frontier model has any business landing. My first thought was that we'd accidentally built the most damning benchmark in the history of our own company, and my second was whether it was too late to go back to doing something honest for a living. Then I remembered the one feature I keep banging on about: we publish every response. So instead of trusting the sad little number, I opened the responses and read them. Every one of the frontier model's failures on the hard problems was the same thing: an empty answer. Not a wrong answer. Nothing at all. Eleven problems, eleven blank pages, each one exactly 4,096 tokens of the model thinking out loud and then having the microphone cut off before it wrote a single line of code. Here's what happened. Our request budget was 4,096 output tokens, a number that's perfectly sensible for "write this short Python function" and perfectly ridiculous for "here's a genuinely hard competitive-programming problem, think it through." The reasoning models did exactly what reasoning models do. They thought. At length. And then they hit the ceiling somewhere in the middle of it and got cut off before writing a line of the solution. So the benchmark wasn't measuring how good the models are at hard problems. It was measuring how good they are at running out of room. We'd built, with some effort, a very reproducible way to gag a model and then mark it absent. I'd love to tell you we spotted this through rigour and foresight. We spotted it because the responses were sitting right there in the file, empty, quietly judging us. Which is, if you squint, exactly the argument for publishing them. A closed benchmark makes this same mistake and ships the number, and everyone nods, because who's going to check. We made the output budget a config setting, gave the hard suite the room it needed, and ran it again. | Policy | Likely-contaminated 20 | Fresh, 2025+ 15 | |---|---|---| | cheapest model only nano | 90.0% | 60.0% | | ideal routing oracle | 100.0% | 93.3% | | always the frontier model | 100.0% | 86.7% | | how often ideal routing reached for the frontier | 0% | 20% | Three things fall out of that, now the models can actually finish. The frontier model's fine: 86.7% on the fresh problems, not the panicked 60% the truncated run reported. The cheap model, though, drops from 90% on the old benchmarks to 60% on the fresh ones, which is the contamination showing through exactly as predicted. And routing earns far more on the fresh set: on the old tasks it buys ten points over the cheap model, on the fresh tasks it buys thirty-three, while reaching for the expensive model a fifth of the time instead of never. The honest asterisk: the fresh set's competitive-programming, which is both newer and harder than grade-school maths, so difficulty and freshness are tangled together here, and the samples are small. It's a first signal, not a verdict. Two things, and I'll keep them honest because you can check both. The contamination point stands. Even with the models given all the room they wanted, the cheap one's thirty points worse on problems it can't have memorised. The "small models are quietly brilliant" story was, in part, small models sitting an exam they'd already seen. The harness point is the one I didn't expect to be writing about, and it's the more useful one. Half of what looks like a model being bad is a benchmark being broken, and you can't tell the difference from a leaderboard. You can only tell by reading the answers. If a routing benchmark, or a routing vendor, won't show you the responses, you've no way of knowing whether their impressive number is a real result or a bug wearing a nice suit. Cost of finding all this out: about $7.47. Four dollars of that was the run that produced the numbers above; the rest was the earlier runs that produced the wrong ones and taught us the lesson. I'm counting the mistakes, because a benchmark that hides its own costs has no business lecturing anyone about hidden numbers. Thanks to deanlee and jugeni for the nudge. The suite grows from here, more fresh problems and harder ones, and every number stays re-gradable offline with omnisbench verify . OmnisBench is built by Fortitude Omnis https://fortitude-omnis.group . We make small, sharp tools, and when we catch ourselves measuring the wrong thing, we write it up rather than quietly fixing the chart.