# I Tested 300+ Models. Then I Killed the Benchmark. > Source: > Published: 2026-07-14 07:03:37+00:00 *Let It Break — part 1* *Tags: #ai #llm #benchmark #postmortem* In May I ran a series called Agent Autopsy. Agents failing — broken packages, forgotten context, cron jobs dying silently — while I was learning what I didn't know I didn't know. Every failure got a post-mortem, because every failure felt like it deserved one. That series didn't end with a finale. It ended when I stopped caring. Somewhere around part eight, broken things stopped feeling like emergencies. Call it tolerance. Call it resilience. Maybe I just got tired. Probably all three. This is what came after: a series about what I kill on purpose, what I leave broken on purpose, and what that buys me. Nothing in this post failed. The benchmark worked perfectly, right up until the evening I killed it. Back then I also wrote about [binning an SPFx agent harness nobody asked for](https://dev.to/vystartasv/a-billion-token-lesson-because-you-can-you-should-56op). In that post I held up the benchmark as the thing people actually wanted. The thing worth keeping. This month I killed the benchmark too. Same lesson. Longer fuse. Ten real-world agent coding tasks — file operations, shell commands, error recovery, data parsing, SQL queries. Every model I could reach through OpenRouter, plus everything I could fit on my own hardware. Max tokens 400, temperature 0.1, pattern-matching scoring, pre-flight verification so a flaky endpoint couldn't fake a zero. The last batch I published put the public dataset at 168 models. The real count — OpenRouter plus local, including batches I never wrote up — passed 300. A 10-model batch cost about $0.10. The 200-call efficiency study cost $0.56. The whole dataset cost less than a takeaway. Cheap to run. Expensive to keep honest. I didn't understand the difference until I was hundreds of models deep. Read my own headlines in order. May: [the best local model scored 50%](https://dev.to/vystartasv/i-tested-6-local-models-on-real-agent-tasks-the-best-scored-50-384o). Days later: five brand new families debuted, none below 75%. Then: two models hit 90%, one for less than a penny. Fifty percent, to a 75% floor, to 90% at sub-penny prices. In weeks. When the punchline of every batch is "they're mostly fine and they're all cheap," the leaderboard has answered its own question. The last finding that actually helped anyone wasn't about model quality at all — [it was that the free tier is a gamble](https://dev.to/vystartasv/10-models-tested-from-816-to-10-the-free-tier-is-a-full-on-gamble-4kfc), 81.6% and 10% in the same batch. The interesting question stopped being "which model can do this." Most of them can. It became "how little scaffolding do I need." A leaderboard can't answer that. Nobody was using it. The posts got reads. A few good comments. But I couldn't find one person making one decision off my numbers. I told myself the audience would arrive once the dataset was big enough. The dataset got big. The audience was still me. Here's the tell I only see in retrospect: the published count stopped at 168, but I kept testing past 300. I was running batches I didn't even bother writing up anymore. A leaderboard nobody read, fed by tests nobody saw — including, eventually, me barely looking at them either. Traction you have to talk yourself into is not traction. Supporting Liverpool teaches you to sit loyally through seasons that are going nowhere. I gave a leaderboard the same loyalty. The leaderboard had not earned it. Staleness, and it was structural. Models land weekly. Every batch was a photograph of a moving train — accurate for about as long as it took to write the post. Keeping the leaderboard honest meant re-running everything, forever, on my own time, for an audience of one. There's a personal irony here. I wrote earlier this year about [running 20 agents on €4.57 a month of infrastructure](https://dev.to/vystartasv/i-built-infrastructure-for-20-ai-agents-that-run-themselves-for-eu457month-1p5l). That efficiency is what kept the benchmark alive past its expiry date. When a zombie project costs pennies to feed and runs itself on cron, killing it requires *noticing*, not budgeting. Cheap automation doesn't just scale the good ideas. And the harness had compromises I'd stopped seeing. The 400-token cap punished verbose-but-correct models. Pattern-matching scored format as much as competence. Fixing that meant more harness. The models were getting better faster than the harness could get fairer. I built scaffolding to measure models. The models outgrew the scaffolding. That's not an engineering failure — that's the ecosystem working as advertised, and me billing myself weekly for refusing to notice. The habit. I still test every model that interests me the day it drops, against tasks I actually care about. That reflex came from the benchmark and outlived it. The findings. I know from data, not vibes: code quality does not equal agent capability, "write efficient code" prompts do nothing for most models, and free tiers charge you in debugging time. The data, archived. If anyone ever genuinely needs it, it can be served through MCP in an afternoon — a query interface, not a leaderboard. No maintenance, no weekly re-runs, value on demand. The thin version of the same idea. The harness does not survive. That's the right way around. The weekend harness died on the whiteboard, where bad ideas are cheap. The benchmark died past 300 models, where they're not. Same disease, later diagnosis: **"is anyone looking for this?" isn't a question you ask once at the start. You ask it every time you're about to maintain something.** Killing it cost one evening. Keeping it cost every week. I did that math embarrassingly late. Next time I kill it at 50 models, not 300. Tolerance, resilience, or just tired? This one was tired — and late. The next posts in this series cover the things I've left broken on purpose. Those are harder to defend. That's why they're worth writing.