Same Model, Different Coding-Agent Harness: Scores Mostly Held, Cost Moved Up to 3x Researchers Yangze Liu and Zhongyi Han of Shandong University found that swapping coding-agent harnesses while holding the model fixed produced no meaningful accuracy advantage, but changed cost per task by up to 3x. Across SWE-bench Verified tasks, Claude Code's first call averaged 17,134 tokens versus 1,382 for mini-SWE-agent, with 16,581 of Claude Code's tokens being harness preamble resent on every step. The authors concluded that "a vendor's own harness on its own model buys no score advantage," though OpenCode trailed by up to 9 points largely because it failed to re-prompt when reasoning exceeded the 32k output cap. Before your coding agent reads a single line of your ticket, it has already sent the model a wall of text. In a new study, that first call carried 17,134 tokens through Claude Code and 1,382 through mini-SWE-agent https://arxiv.org/html/2610.04433v1 on average, on the same model and the same 45 tasks. The paper is What Does a Harness Buy? Tokens, Mostly https://arxiv.org/abs/2610.04433 , by Yangze Liu and Zhongyi Han of Shandong University. It went up on arXiv on October 3, and VentureBeat covered it on October 23 https://venturebeat.com/orchestration/a-coding-agents-harness-can-triple-its-cost-without-making-it-more-accurate-study-finds alongside a second study from Berkeley that points the same way. The question it asks is one you face every time you pick a coding agent: if you keep the model fixed, how much does the agent around it change the result? Its answer, in the authors' words https://arxiv.org/html/2610.04433v1 : "A vendor's own harness on its own model buys no score advantage." The cost per task, though, varied up to 3x https://arxiv.org/abs/2610.04433 . A harness is everything around the model: the system prompt, the tool set and how the agent manages context. The paper held the model fixed and swapped three harnesses on SWE-bench Verified https://www.swebench.com/ tasks, run offline in containers: According to VentureBeat https://venturebeat.com/orchestration/a-coding-agents-harness-can-triple-its-cost-without-making-it-more-accurate-study-finds , the models were two local open-weight Qwen models Qwen3.6-35B-A3B and Qwen3.8-27B , three API models DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview , and Claude Opus 5 inside Claude Code as an anchor. The repository https://github.com/YangzeLiu/what-does-a-harness-buy ships all 8,032 trials as a CSV, MIT-licensed code, and a reproduce.sh that rebuilds the paper's numbers on a CPU in under a minute. Same engine, three harnesses. The study measured what the extra baggage costs. On the 447-task pool, Claude Code and mini-SWE-agent scored within five points of each other https://arxiv.org/abs/2610.04433 , and on the two Qwen models the harnesses were statistically equivalent within a 5-point margin https://venturebeat.com/orchestration/a-coding-agents-harness-can-triple-its-cost-without-making-it-more-accurate-study-finds . On a 45-task hard subset, switching harnesses flipped about 13% of tasks https://arxiv.org/abs/2610.04433 between pass and fail. Rerunning the same harness flipped about as many. OpenCode was the one harness effect that cleared the noise, trailing by up to 9 points https://arxiv.org/abs/2610.04433 . Much of that came from one behaviour. When a model's reasoning ran past the 32k output cap mid-step, OpenCode did not re-prompt and the trial usually ended with no patch https://arxiv.org/html/2610.04433v1 . Claude Code re-prompts. Excluding those cut-off runs roughly halves OpenCode's gap, per the paper. The authors are careful about what that doesn't prove. Their run table https://arxiv.org/html/2610.04433v1 shows most cells on the 447-task pool have a single run. Their limitations section notes it's one benchmark and that the closed model, Opus 5, only ran inside Claude Code. So nobody knows yet whether a frontier closed model shows the same small gaps. They also show that 45 tasks detect a 13-point gap only about half the time https://arxiv.org/abs/2610.04433 . That's worth remembering the next time a vendor quotes a 3-point harness win. Cost is where the harnesses split. On GLM-5.3-Flash, a trial cost 1.81 CNY in Claude Code, 0.97 in mini-SWE-agent and 0.56 in OpenCode https://arxiv.org/html/2610.04433v1 . The preamble explains much of it: of Claude Code's 17,134 first-call tokens, 16,581 were harness preamble https://arxiv.org/html/2610.04433v1 , against 829 for mini-SWE-agent. That preamble is resent on every step. But the preamble alone doesn't decide it. On DeepSeek-V4-Flash the order flipped: 0.98 CNY for Claude Code, 1.02 for mini-SWE-agent https://arxiv.org/html/2610.04433v1 . VentureBeat reports https://venturebeat.com/orchestration/a-coding-agents-harness-can-triple-its-cost-without-making-it-more-accurate-study-finds that mini-SWE-agent took 141 model calls per trial there, against Claude Code's 104. A lean harness that takes more steps can lose its head start. Caching softens all of this without changing the ranking: 96% to 99% of input tokens hit the provider's prompt cache https://arxiv.org/html/2610.04433v1 in every arm. The paper's summary is the line to keep: "The numbers to carry between vendors are the first call, the growth per step, and the step count." You can see why with a toy model. Each step resends the preamble plus the history so far: python def input tokens preamble, growth, steps : step k resends the preamble plus k steps of history return sum preamble + growth k for k in range steps big = input tokens 16 581, 1 500, 100 small = input tokens 829, 1 500, 100 With an assumed 1,500 tokens of history per step, that's 9,083,100 against 7,507,900 input tokens over 100 steps. The 1,500 is my placeholder, not a number from the paper. The point is the shape: over a long run, history growth and step count can matter as much as the preamble. First, network access. In web-enabled runs, 312 of 439 DeepSeek and Qwen trials 71% fetched the upstream fix from GitHub https://arxiv.org/html/2610.04433v1 , so the authors threw those runs out and moved the offline condition from the prompt to the network layer https://arxiv.org/html/2610.04433v1 . If you benchmark agents on public repos, a "don't browse" line in the prompt isn't enough. Second, thinking defaults. Turning thinking off for Qwen3.6 dropped mini-SWE-agent's pass rate by 11.1 points https://arxiv.org/html/2610.04433v1 , while Claude Code and OpenCode gained a little. An older Claude Code release that didn't enable extended thinking by default passed 34 of 45 hard tasks against 35 to 36 for newer ones, at 0.44x the cost https://arxiv.org/html/2610.04433v1 . The harness and its defaults move together, so when you compare two agents, write down both. The Berkeley study lands near the same place. VentureBeat reports https://venturebeat.com/orchestration/a-coding-agents-harness-can-triple-its-cost-without-making-it-more-accurate-study-finds that HarnessTax https://harnesstax.github.io/ , from UC Berkeley's Sky Computing Lab and Arena, ran seven models across Claude Code, Codex CLI https://github.com/openai/codex and Pi https://github.com/badlogic/pi-mono . Pass rates moved a few points. Inference cost moved up to fivefold, and Claude Code's initial context was more than 10 times Pi's. A paper on SWE-bench is a starting point. Your repo, your tickets and your model are what you pay for, so measure there: same model, same ticket, two harnesses, and record cost per task, not only pass or fail. Run each side twice, because the study shows a rerun moves results about as much as a harness swap. I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex, OpenCode, Aider and any command-line agent you add https://github.com/hermes-hq/hermes-ide each in its own terminal, and ⌘N starts each task on its own hermes/