{"slug": "how-to-read-a-coding-agent-benchmark-without-getting-sold", "title": "How to read a coding-agent benchmark without getting sold", "summary": "A nine-author study led by Fan et al. held the underlying model and execution loop fixed while varying three coding-agent harness components — planning, action space, and context management — across 176 matched configurations, four models, and two benchmarks (SWE-Bench Verified and Terminal-Bench 2.1). The researchers found that context management mainly prevents failure when the context window overflows rather than improving reasoning, that explicit planning helps weaker models but becomes a cost saver with a small accuracy penalty on stronger ones, and that predefined tools help models weak at bash while bash-capable models do better and cheaper with a shell alone. The work argues that published benchmark scores reflect the whole agent, not the model, and that every added tool is a claim requiring testing.", "body_md": "*A nine-author study this week pulled a coding agent apart into its components and measured each one across 176 configurations. The findings are less exciting than any vendor slide and more useful than all of them, and they hand the buyer four questions no benchmark answers.*\n\nEvery coding-agent pitch you have seen this year has a number on it. SWE-Bench Verified, some\n\npercentage, up and to the right, usually next to a model name. The implication is that the number\n\nbelongs to the model, and that if you buy the model you get the number.\n\nA commenter on this week's Hacker News thread about the harness study put the problem better than\n\nthe paper's abstract does. If Car A is faster than Car B, it is not necessarily the engine. It could\n\nbe the tyres, the gearbox, the weight, the driver. A coding agent is a car. The model is the engine.\n\nThe harness, meaning the loop around the model that decides what it sees, what it can do and when\n\nit stops, is everything else. And the number on the slide is a lap time for the whole car, measured\n\non a track you do not drive on.\n\nNine researchers at Fan et al. did the thing nobody selling these tools has an incentive to do. They\n\nheld the model fixed, held the execution loop fixed, and varied three harness components one at a\n\ntime: planning, action space, and context management. Four models, two benchmarks (SWE-Bench Verified\n\nand Terminal-Bench 2.1), 176 matched configurations, five context-management strategies, four\n\ncontext-window budgets. Then they looked at the trajectories, not just the scores, to see what each\n\ncomponent actually changed about how the agent behaved.\n\nThe findings are almost aggressively unglamorous, which is how you know they are worth something.\n\nContext management, the machinery that decides what to throw away as the conversation fills up,\n\nmatters more the tighter the context budget, and most of its benefit comes from one thing: not\n\nfalling over when the window overflows. It does not make the agent smarter. It lets the agent keep\n\ngoing. The strongest strategy in their comparison was the boring one, a rule-based pass that\n\ndeletes obviously stale material before any model-based summarisation runs. And the clever\n\naddition everyone builds, making elided content recoverable so the agent can go back and fetch it,\n\nturned out to be machinery the models rarely used and which yielded no accuracy gain.\n\nPlanning, meaning an explicit plan-first step, changes role depending on the model. For weaker\n\nmodels it is an accuracy scaffold. For stronger ones it stops helping accuracy and becomes a cost\n\nsaver, with a small decrease in success rate as the price. Another commenter connected this to\n\nsomething in the Claude Code changelog: the built-in todo and task-tracking tools were switched off\n\nby default on the newest model generations. The vendor, in other words, appears to have measured the\n\nsame thing.\n\nAction space, meaning whether the agent gets a menu of predefined tools or just a shell, splits the\n\nsame way. Predefined tools raise success rates for models that are weak at driving bash. Models that\n\nare good at bash do better and cheaper with bash alone, most clearly on command-line-shaped tasks.\n\nThe paper does not define \"bash-capable\" crisply, which a commenter rightly flagged, but the\n\ndirection is unambiguous: every tool you add beyond the shell is a claim that needs testing, not a\n\nfree improvement.\n\n*What each harness component does, by model strength, per the study's abstract. Read the row for the model you actually run.*\n\nThe trajectory analysis is the part a buyer should care about most. Context management extended\n\nhow long the agent could keep working without substantially changing what it did. Planning changed\n\nwhere trajectories stopped. Action space changed the granularity at which code got written. None of\n\nthe three made the engine better. They changed the gearbox, the tyres and the driver, and the lap\n\ntime moved accordingly.\n\nPut the study next to what the practitioners were saying this week and a picture forms.\n\nTheo Browne's video argued, with some heat, that people who cannot feel the difference between\n\nmodel generations are prompting badly, and his most useful idea was a picture of the distribution:\n\nevery model has a ceiling, which is what the demos show, and a floor, which is what you hit at two\n\nin the afternoon on a boring task. Frontier models mostly raise the floor. ThePrimeagen, the same\n\nweek, left a current-generation model on a trivial colour bug, came back forty minutes later, and\n\nfound it had spent 330 million tokens and 118 dollars reading the same file over and over. That is\n\na floor.\n\nA benchmark number is a ceiling measurement of one car on one track. It tells you nothing about the\n\nfloor, and the floor is where your money goes. And the study tells you the floor is mostly a\n\nharness property: whether the loop notices it is stuck, whether the context gets cleaned before it\n\noverflows, whether the agent has a shell or a menu, whether there is a plan and whether the plan is\n\nworth its cost for the model you actually run.\n\nSo when the next vendor slide arrives, the number is not the question. These are.\n\nWhich harness produced this number, and can I see it? If the answer is \"our proprietary agent\n\nruntime\", you are buying a car and being told the horsepower. Ask for the loop: what the model\n\nsees, what it can do, how context is managed, when it stops.\n\nWhat does it do when it is stuck? Ask for the failure trajectories, not the success ones. A good\n\nvendor has them and is proud of them. Ask specifically what happens at context overflow and what\n\nhappens after the tenth identical tool call. The study says that is where the benefit of the whole\n\ncontext-management apparatus lives.\n\nWhat does it cost per success at the floor, not per success on the benchmark? Your workload is not\n\nSWE-Bench. Take twenty of your own boring tasks, run them, and divide dollars by successes. Include\n\nthe runs you killed. The study's own finding, that stronger models do better and cheaper with fewer\n\ntools, is a hypothesis you can test on your codebase in an afternoon.\n\nWhich components would I turn off? This is the question the paper actually equips you to ask.\n\nIf your model is strong, the plan step might be a cost centre. If it is bash-capable, the tool menu\n\nmight be a drag. If your context budget is generous, the elaborate recoverable-summary system might\n\nbe doing nothing. A harness with fewer components that you understand beats one with more that you\n\ndo not, for the same reason a car you can service beats one you cannot.\n\n*A benchmark measures the whole car once, at its ceiling. A buyer needs the floor, per component, on their own track.*\n\nSince the third question is the one that matters and the one a vendor cannot answer for you, here is\n\nthe protocol we use, which costs an afternoon and a modest API bill.\n\nPick twenty tasks from your own recent history. Not the interesting ones. The ones that came in as\n\ntickets and got done without anyone remembering them: a null check, a copy change, a small\n\nmigration, a flaky test, a dependency bump that broke something. Ten of them should be the kind of\n\ntask an intern would finish before lunch. Ten should be the kind that looks trivial and turns out to\n\ntouch four files. Write each one down as a ticket, the way it was actually written, with the same\n\nmissing context.\n\nRun each task through the candidate agent with its default harness, on a clean checkout, with a\n\nfixed budget of tokens or minutes, and walk away. Do not steer. Steering is what the demo does and\n\nit is what you will not have time to do at volume. When the budget runs out or the agent stops,\n\nrecord three things: did it produce a change a reviewer would accept, how many tokens did it use,\n\nand did it at any point loop, meaning repeat an action it had already taken with the same result.\n\nThen divide the money by the accepted changes. That number, dollars per accepted change on your own\n\ndull work, is the only benchmark that will predict your bill. Run the same twenty through a second\n\ncandidate and you have a comparison that no leaderboard offers. Run them again with the plan step\n\ndisabled, or the tool menu replaced with a shell, and you have reproduced the study's method on the\n\nonly codebase you care about. The loops column is the floor made visible; a candidate that looped\n\non three of twenty will loop on fifteen percent of your work forever, and no ceiling justifies that.\n\nIt was run on a particular set of models that, as one commenter complained, did not include the\n\ncurrent frontier or the strongest open-weight options. The definitions are looser than they should\n\nbe. The benchmarks are the benchmarks, with all the training-set contamination questions those\n\ncarry. None of that changes the shape of the result, which is that the harness is a first-class\n\nvariable and the leaderboards treat it as noise.\n\nThe engine matters. Buy a good one. But you are going to spend the next year driving the car, and\n\nthe car is the part you were never shown.\n\n*Originally published on the [Levelbrook playbook](https://ai.levelbrook.com/playbook/how-to-read-a-coding-agent-benchmark-without-getting-sold/). Levelbrook is a principal-led Rails and AI-systems consultancy; the playbook is where we write down what we see.*", "url": "https://wpnews.pro/news/how-to-read-a-coding-agent-benchmark-without-getting-sold", "canonical_source": "https://dev.to/levelbrook/how-to-read-a-coding-agent-benchmark-without-getting-sold-2pf1", "published_at": "2026-09-21 14:19:06+00:00", "updated_at": "2026-09-21 14:32:32.491967+00:00", "lang": "en", "topics": ["ai-agents", "ai-research", "large-language-models", "developer-tools"], "entities": ["Fan et al.", "SWE-Bench Verified", "Terminal-Bench 2.1", "Claude Code", "Hacker News"], "alternates": {"html": "https://wpnews.pro/news/how-to-read-a-coding-agent-benchmark-without-getting-sold", "markdown": "https://wpnews.pro/news/how-to-read-a-coding-agent-benchmark-without-getting-sold.md", "text": "https://wpnews.pro/news/how-to-read-a-coding-agent-benchmark-without-getting-sold.txt", "jsonld": "https://wpnews.pro/news/how-to-read-a-coding-agent-benchmark-without-getting-sold.jsonld"}}