Bakeoff: More thinking isn’t (always) better

A Bakeoff benchmark of 5 coding agents across 5 codebases found that raising effort levels roughly doubled the amount of code changed but only 2 of 5 models solved more tasks, with Fable 5.1 performing worst at its high default setting. Opus 5.5 at high effort was the only setting to fix the hardest bug in all 3 of 3 runs, while Sonnet 5.5 and Fable 5.1 solved fewer tasks at high than at low and Haiku 5.5 stayed flat from medium to high. At Claude Code defaults, Opus 5.5's medium setting reached an 88% success rate at $1.07 per run with a median time of 3m10s, versus Fable 5.1's high default at 75% for $3.56 per run and 6m28s, and high effort lengthened mean diffs from 126 lines at low to 281 at high.

Turning coding agents up to high effort roughly doubled how much code they changed, and only 2 of 5 models solved more tasks with it. Fable 5.1’s default was its worst setting. You’ve probably noticed the effort aka “thinking” or “reasoning” setting in your coding agent of choice. If you’ve thought about it at all, you set it by how hard the problem feels. A brief primer: thinking is the scratch work a model writes for itself before it answers, and the effort level sets roughly how much of it the model does. Here’s the effort level each agent ships with by default: Messy, right? In my case, I didn’t realize that GPT-6.1 Sol was set to “low” thinking for its first 3 weeks in Bakeoff https://fairbake.dev/bakeoff , which hurt when doing comparisons to Opus. Ouch. I ran 5 agents at up to three effort levels on real tasks bug fixes and new features from 5 codebases, graded by tests the agents never saw. Each agent ran each task 2 or 3 times at each level. The 5 codebases span several languages and shapes: 2 full-stack JavaScript web apps, 2 Python data pipelines, and a Rust tool that intercepts a client’s network traffic. Here are the results full table in the appendix : Opus 5.5 and GPT-6.1 Sol got better every time I turned them up. Opus at high was the only setting to fix the hardest bug every time 3 of 3, against 1 of 3 at medium and max, and 0 for every other model , and Sol at high built a multi-user feature it missed every time at low. The other 3 just got more expensive. Sonnet 5.5 and Fable 5.1 solved fewer tasks at high default for Fable then at low, and Haiku 5.5 stayed flat from medium to high. At each model’s Claude Code default, Opus 5.5 beat Fable 5.1 on solves, cost, and time: - Opus 5.5 ’s medium default had a success rate of 88% for $1.07 a run with a median time of 3m10s - Fable 5.1 ’s high default had a success rate of 75% for $3.56 a run with a median time of 6m28s Anthropic’s own write-up on effort https://claude.dev/blog/spending-your-effort/ found that effort doesn’t rescue a model that picks the wrong approach, which is what sank Fable on the hardest bug, where it missed the same cause on every run. One side effect of increased coding agent usage is that the bottleneck in developing software is no longer coding speed, but instead human review to ensure software meets its requirements. Several engineering leaders I’ve talked to over the last few months have named human code review as the process that agents strain most. Code review was already no one’s favorite task before coding agents. High effort levels make the diffs the code a reviewer has to read 2.2x longer: For runs with a working solution the mean diff goes from 126 lines at low to 281 at high. If you assume your engineers are reviewing 300 lines of code per hour and have 10 agent PRs a week to review, they’re going to spend 5 extra hours a week on just reviewing diffs from agents that spent their time writing longer solutions to problems that could’ve been solved with fewer. I ran one last experiment on effort levels, giving Opus 5.5 three chances to run at “max” effort level on the hardest bug in the set. The result was that Opus did worse at max than it did at high: My best guess on this failure is overthinking, which Claude Code’s own docs https://code.claude.com/docs/en/model-config warn about at max: it rewrote far more code than at high a median of 664 lines against 140 . What I’d set If I were setting standards for a team today, here’s what I’d do: - Set effort level defaults per model. - Don’t default to max. - Fable 5.1 : Set to low, if you use it at all. - Sonnet 5.5 : Set to low. Gets an 88% success rate at $0.31 per run, under a quarter of the $1.39 Opus at high charges for more solves. - Opus 5.5 / GPT-6.1 Sol: Set to low for routine work and high for hard tasks. Both tools let admins set an org-wide default to roll this out. What I don’t know: the tasks I used aren’t yours, and your effort curves will be different. If you want to see yours, book 30 minutes with me at fairbake.dev https://fairbake.dev/ and I’ll get you set up with a bake. Thanks to Alex Monroe and Bryan Chen for contributing code and Jackie Hyman https://jacquelinediamond.net/ for editing Appendix Solve rate, time, and cost by effort level | Model | Runs per level | Low | Medium | High | |---|---|---|---|---| | Haiku 5.5 | 21 | n/a | 69% · 2m49s · $0.047 | 69% · 3m45s · $0.068 | | GPT-6.1 Sol | 21 | 75% · 2m38s · $0.18 | 79% · 3m58s · $0.22 | 88% · 5m44s · $0.33 | | Sonnet 5.5 | 21 | 88% · 1m33s · $0.31 | 81% · 1m29s · $0.35 | 81% · 2m50s · $0.70 | | Opus 5.5 | 21 | 79% · 1m49s · $0.58 | 88% · 3m10s · $1.07 | 96% · 4m18s · $1.39 | | Fable 5.1 | 16 | 88% · 4m07s · $2.33 | 88% · 5m45s · $3.11 | 75% · 6m28s · $3.56 | Solve rate and cost is averaged over the 8 tasks, time is calculated as the median across tasks