The same bug fix costs 0.4¢ or $2, depending on which coding agent you ask

Every company buying coding tools and agentic workflows right now is deciding on vibes. I spent the last month building a private eval on code no model has seen. Here is what building it taught me, and what I found out.

Quick context, since last week’s post was the teaser: Bakeoff takes real bug-fix or feature commits from my own repos, hides the tests that prove the fix or new feature worked, hands a prompt and the broken tree to a coding agent, and grades whatever comes back against the hidden tests.

By the numbers:

• 277 recorded runs across 10 entrants (harness x model combinations), spanning 10 models and 3 harnesses (Claude Code ,Codex CLI , andpi ), at an approximate cost of $120 at list API rates.
• 7 bug-fix or feature commits, each with a handcrafted prompt (a “challenge”), mined from 3 apps I wrote and run: two Express web apps and a Python agent that runs as a cron job. None of this code is public, so no model has seen the fixes.
• Bakeoff repo: 28 merged PRs, 102 commits, and about 7k lines each of design docs and Go code.

I might have hand-written 10 lines of code in total. My month instead went to thinking through new features, iterating on designs my agent drafted, watching how the implementation behaved, and reviewing code before merge. This isn’t a new way of working for an experienced engineer. It’s roughly the same split I’ve had for the last five years, minus the part where I typed the code.

Building Bakeoff #

What kept quality high

Not writing the code only works if something else catches the mistakes. Agents thrive with guard rails, ideally deterministic ones. Bakeoff ended up with 78 unit tests, gofmt running on every push, CI that strictly checks configuration YAML, and an AGENTS.md that standardized the rest: design docs before code, a formalized roadmap, branching and PR rules, and manual test steps written down so they can be re-run.

The same instinct applies to the agent eval itself. Every run keeps its full session transcript and output tree on disk, which has paid for itself many times over in debugging broken harnesses, and it’s where questions like “do agents that write their own tests do better?” get answered.

I picked tools that made those guard rails cheap. Go is a great language for agents - type safe, simple, batteries included, and lots of examples in the training data. It also feels nice after years of dealing with legacy Java and C++ codebases.

Where the agent needed me

The above sounds nice, but none of my plans have ever survived first contact with a coding agent. Here’s some of the taste I added during development, mostly with Claude Code and Fable 5.1:

• Preventing the answer key from leaking - Claude added arevert.paths field to each challenge, listing which files in the fix commit to roll back before handing the tree to the agent under test. That fails dangerously: forget one path and the agent sees the fix. I flipped it to start from the pre-fix commit and list only the tests to keep.
• Rejecting a requirement for “guard” tests - The initial design required every challenge to name tests that catch an over-correction, like an agent that fails every request so the “reject bad requests” test goes green. I pushed back. It’s a nice way of categorizing failures, but it would have significantly slowed down the rate of adding challenges, which mattered more.
• Deleting a check that only caught one bug - A side project of mine had a test failing at HEAD because it comparedtoday() to a fixed date. Claude’s design wanted to detect that specific mistake and reject such tests from challenges. That check added complexity without solving the general problem of time-dependent tests, so I removed it.
• Deleting code history embedded in comments - When I corrected something, Claude would leave both the original rationale and the correction in the comments. That’s duplicative and harder to read, and the commit history already records the mistake.

Software is still hard

The agent sandboxing problem was harder than I’d planned for. pi reads instruction files from parent directories, even when told to stay in its work directory. Claude Code’s default sandboxing broke uv, which needs a cache directory to work. Building a consistent set of adapters for each agent harness was a grind of small details in config files, environment variables, and flags. And one of those details really mattered (see Results).

Results #

Once the harnesses stopped fighting me, the runs started giving actionable intelligence. Note that because entrants are harness × model combinations, Fable vs Astra is also Claude Code vs Codex. GLM-5.2 was run via OpenRouter. Four insights stand out.

The same bug fix costs 2 cents or $2, depending on who does it. The spread in cost between green solutions is over 80x: GPT-5.6 Luna in Codex and GLM-5.2 in pi at the bottom, Opus 5 and Fable 5.1 in Claude Code at the top. The newest frontier entrants use 10% fewer output tokens than their predecessors, but their per-token rate more than eats the difference.

This cost difference is a significant problem on the work everyone wants to automate: an attempt on every failing CI build, flaky test, unstaffed ticket, and failed request. A thousand of those a day is about $1,700 a day, or $50k a month.

Databricks saw the same thing on their own multi-million-line codebase in July (link): an open-weight model matched their frontier model’s completion rate at two thirds the cost per task.

Speed varies by entrant significantly, and it sorts by harness, not model [see update below for changes to this]. Across an 80x spread in cost, the slowest entrant takes only 2x as long as the fastest: 1m33 for GPT-5.6 Sol to 3m12 for Opus 5. Every Codex entrant finished ahead of every Claude Code entrant. Within Claude Code, Fable 5.1 beats Opus 5 by about 20%, and the three Codex models are within 25% of each other.

The frontier models write tests without being asked. GPT-6 Astra touched the test suite on 12 of 14 runs and Fable 5.1 on 6 of 14. Every other entrant did 3 times or fewer. It’s the first behavioral split in the data, and it didn’t change who solved what - yet. These challenges are small enough that a careful diff gets you there. The harder ones are next.

A sandbox setting prevented a harness from running tests. Codex’s default sandbox blocks the test server from starting by preventing it from binding a port, so on 6 of 7 challenges its models never saw any integration tests go green. They improvised with one-off scripts instead, and mostly got away with it. But the only two failures in the data are both Codex, and both were slips integration test runs would have caught. It took two full sets of runs and a read through the transcripts to notice. A healthy success rate hid an agent working with one hand tied behind its back.

Three rules for your team #

Confirm your agents can get your test suite passing before comparing anything. Sandboxes, missing caches, and blocked ports all handicap an agent silently. If it starts writing one-off scripts or installing dependencies instead of running your tests, something is in the way.

Run the cheap model first, escalate on a red test. GPT-5.6 Luna passed 14 of 15 runs and GLM-5.2 passed all 17, each for a few cents a task against $1.73 for the most expensive entrant.

Keep every session transcript. The port-binding failure above was invisible in the scores and obvious in the logs. The questions you’ll want to ask next month are only answerable if the sessions are on disk.

What’s next for Bakeoff #

The current challenges are as unrepresentative as they’ll ever be and the numbers are only a few runs deep. The roadmap is about fixing both, starting with the items that matter most if you’re picking tools for a team:

Broader and harder tasks. Fixes or new features that span domains, services, and/or repos, where the symptom or visible output is far from the relevant code and a plausible wrong implementation is likely. This is where correctness rankings separate frontier models and where you start to be able to trust your most critical business systems to agents. Maybe even without code review!

What are you actually paying for? Three experiments on the cost axis. A survey of cheaper open-weight models, to see what you’ve been missing if you’ve only run Codex or Claude. The same model in different harnesses, to find out whether pi’s lean prompts are a token saving or a quality loss. And thinking levels, which most teams leave at default without knowing what it costs or buys them.

Does your AGENTS.md matter? Same model, same tasks, different instruction files: coding style rules, SDLC rules, none at all. Teams argue about these for hours. The eval can settle it in one set of challenge evaluations.

Bring me your commits #

If your team is picking agentic tooling on vibes, bring three recent commits to a 30-minute call: bug fixes or features, each with the test that proved it. I’ll walk through what a challenge evaluation on each would measure, what it would take to run inside your environment, and what the results above suggest before you run anything. Message me on LinkedIn or send me an e-mail. Thanks to everyone who reached out since the teaser, and especially those who’ve met me directly. Those chats are shaping which way Bakeoff evolves.

Update: Opus 5.5, GPT-6 Luna, and GPT-6 Sol #

I ran these new models through the same seven challenges twice. Opus and Luna are the most interesting new models because of where they sit on the intelligence/cost/speed curve:

Opus 5.5: Clean pass on all tasks, like Opus 5, at a third of the cost and 2.5x the speed: $0.53 and 1m15s per task on average, against $1.61 and 3m12s. Most of the saving is output: it writes half as many tokens as Opus 5, and each one costs 20% less. That puts it next to GPT-6 Astra on price and ahead of every entrant except GPT-6 Luna on speed.

GPT-6 Luna: Solved 12 of 14 (GPT-5.6 Luna: 14 of 15), at a fifth of the cost and nearly 3x the speed: $0.004 and 40s per task, against $0.020 and 1m55s. It is now the cheapest and fastest entrant by a long shot. One miss was the same kind as GPT-5.6 Luna’s: the right fix in the wrong place, which requests to the running app would have caught. The other was new. It read three files, reported “Updated the [website] pagination loop,” and hadn’t changed anything.

GPT-6 Sol: Solved 13 of 14 at $0.15 and 1m44 per task, a bit cheaper and a bit slower than GPT-5.6 Sol, though Codex runs it at a higher default reasoning level. Nothing really new here.

What held: Start with the cheap model and escalate on a red test. Luna’s phantom fix is a case for trusting the test over the agent’s own summary. Test writing still looks like mostly a trait of the model line, not the release: GPT-6 Luna wrote its own tests on 0 of 14 runs, while GPT-6 Astra, from the same generation, did on 12 of 14. Every failure in the data is still a Codex one.

What changed: Due to Luna, the cost spread between green solutions is now even bigger - 400x rather than 80x, the slowest entrant takes 5x as long as the fastest rather than 2x, and speed no longer sorts by harness.

Appendix #

List of entrants

Results caveats

Costs are measured as real OpenRouter spend for GLM-5.2 and list API prices from Claude Code and Codex. Timing measures wall-clock time, including the vendor’s serving latency and the harness’s turn loop as well as the model. The number of runs per entrant, 10 to 17, differs because Fable 5.1 and GPT-6 Astra were added after they were released, and I stopped re-running Opus 5 and Sol on older challenges after their successors were added. The charts and the entrant table use the 96 runs of the first full sweep. The first 49 runs were under an earlier format, the next 49 are the update section, and the last 83 are the Codex re-run mentioned below. Every entrant has at least one run on every challenge. All Codex runs in this post, including the update, were restricted from binding ports. Since publishing, Codex can bind localhost and run its integration tests. Re-running every Codex entrant on the Node challenges barely moved outcomes and left the cost ordering unchanged.