Before your coding agent reads a single line of your ticket, it has already sent the model a wall of text. In a new study, that first call carried 17,134 tokens through Claude Code and 1,382 through mini-SWE-agent on average, on the same model and the same 45 tasks.
The paper is What Does a Harness Buy? Tokens, Mostly, by Yangze Liu and Zhongyi Han of Shandong University. It went up on arXiv on October 3, and VentureBeat covered it on October 23 alongside a second study from Berkeley that points the same way. The question it asks is one you face every time you pick a coding agent: if you keep the model fixed, how much does the agent around it change the result?
Its answer, in the authors' words: "A vendor's own harness on its own model buys no score advantage." The cost per task, though, varied up to 3x.
A harness is everything around the model: the system prompt, the tool set and how the agent manages context. The paper held the model fixed and swapped three harnesses on SWE-bench Verified tasks, run offline in containers:
According to VentureBeat, the models were two local open-weight Qwen models (Qwen3.6-35B-A3B and Qwen3.8-27B), three API models (DeepSeek-V4-Flash, GLM-5.3-Flash and HY4-Preview), and Claude Opus 5 inside Claude Code as an anchor. The repository ships all 8,032 trials as a CSV, MIT-licensed code, and a reproduce.sh that rebuilds the paper's numbers on a CPU in under a minute.
Same engine, three harnesses. The study measured what the extra baggage costs.
On the 447-task pool, Claude Code and mini-SWE-agent scored within five points of each other, and on the two Qwen models the harnesses were statistically equivalent within a 5-point margin. On a 45-task hard subset, switching harnesses flipped about 13% of tasks between pass and fail. Rerunning the same harness flipped about as many.
OpenCode was the one harness effect that cleared the noise, trailing by up to 9 points. Much of that came from one behaviour. When a model's reasoning ran past the 32k output cap mid-step, OpenCode did not re-prompt and the trial usually ended with no patch. Claude Code re-prompts. Excluding those cut-off runs roughly halves OpenCode's gap, per the paper.
The authors are careful about what that doesn't prove. Their run table shows most cells on the 447-task pool have a single run. Their limitations section notes it's one benchmark and that the closed model, Opus 5, only ran inside Claude Code. So nobody knows yet whether a frontier closed model shows the same small gaps. They also show that 45 tasks detect a 13-point gap only about half the time. That's worth remembering the next time a vendor quotes a 3-point harness win.
Cost is where the harnesses split. On GLM-5.3-Flash, a trial cost 1.81 CNY in Claude Code, 0.97 in mini-SWE-agent and 0.56 in OpenCode. The preamble explains much of it: of Claude Code's 17,134 first-call tokens, 16,581 were harness preamble, against 829 for mini-SWE-agent. That preamble is resent on every step.
But the preamble alone doesn't decide it. On DeepSeek-V4-Flash the order flipped: 0.98 CNY for Claude Code, 1.02 for mini-SWE-agent. VentureBeat reports that mini-SWE-agent took 141 model calls per trial there, against Claude Code's 104. A lean harness that takes more steps can lose its head start.
Caching softens all of this without changing the ranking: 96% to 99% of input tokens hit the provider's prompt cache in every arm. The paper's summary is the line to keep: "The numbers to carry between vendors are the first call, the growth per step, and the step count."
You can see why with a toy model. Each step resends the preamble plus the history so far:
def input_tokens(preamble, growth, steps):
return sum(preamble + growth * k for k in range(steps))
big = input_tokens(16_581, 1_500, 100)
small = input_tokens(829, 1_500, 100)
With an assumed 1,500 tokens of history per step, that's 9,083,100 against 7,507,900 input tokens over 100 steps. The 1,500 is my placeholder, not a number from the paper. The point is the shape: over a long run, history growth and step count can matter as much as the preamble.
First, network access. In web-enabled runs, 312 of 439 DeepSeek and Qwen trials (71%) fetched the upstream fix from GitHub, so the authors threw those runs out and moved the offline condition from the prompt to the network layer. If you benchmark agents on public repos, a "don't browse" line in the prompt isn't enough.
Second, thinking defaults. Turning thinking off for Qwen3.6 dropped mini-SWE-agent's pass rate by 11.1 points, while Claude Code and OpenCode gained a little. An older Claude Code release that didn't enable extended thinking by default passed 34 of 45 hard tasks against 35 to 36 for newer ones, at 0.44x the cost. The harness and its defaults move together, so when you compare two agents, write down both.
The Berkeley study lands near the same place. VentureBeat reports that HarnessTax, from UC Berkeley's Sky Computing Lab and Arena, ran seven models across Claude Code, Codex CLI and Pi. Pass rates moved a few points. Inference cost moved up to fivefold, and Claude Code's initial context was more than 10 times Pi's.
A paper on SWE-bench is a starting point. Your repo, your tickets and your model are what you pay for, so measure there: same model, same ticket, two harnesses, and record cost per task, not only pass or fail. Run each side twice, because the study shows a rerun moves results about as much as a harness swap.
I build Hermes IDE, so weigh this with that in mind. Hermes runs Claude Code, Codex, OpenCode, Aider and any command-line agent you add each in its own terminal, and ⌘N starts each task on its own hermes/<name> branch in its own worktree. The sidebar shows spend per session: the cost an agent reports, or an estimate from its transcript marked as estimated, with token totals. That gives you a cheap way to set the same ticket on two agents and compare what each one burned. I haven't run the paper's setup through it myself.
Codex and Claude Code side by side, each in its own hermes/ worktree, with status, context use, cost and branch per session.
Hermes is free to download, source-available (BSL 1.1) and runs on macOS, Windows and Linux. It isn't affiliated with Nous Research, Anthropic, OpenAI or any agent vendor named here. Download it here.
Have you compared two harnesses on the same model and seen a cost gap like this, or did a richer harness pay for itself on your codebase? Tell me in the comments.
The paper's real lesson is that you can't pick an agent from its score alone: you need the first-call size, the growth per step and the step count for your own tasks. The LLM Observability Pocket Guide covers the tracing and eval tools that get you those numbers, and how to choose between them for your team. If you'd rather keep two agents running side by side while you measure, Hermes IDE is free to download.