Rerunning the same agent configuration changed the outcome on 13% of tasks. Swapping the harness — the system prompt, tool set, and context management wrapped around the model — changed the outcome on 13% of tasks too.
That is the central result of "What Does a Harness Buy? Tokens, Mostly," a preprint by Yangze Liu and Zhongyi Han at Shandong University, published as arXiv:2610.04433 on October 3, 2026. It has not been peer reviewed. It covers one benchmark family and three production harnesses, so it is a data point rather than a law. But it asks a question most harness comparisons avoid: when you change the wrapper and keep the model fixed, how much of the score change is real?
The authors put five models through three harnesses: Claude Code, mini-SWE-agent, and OpenCode. Claude Code is the heavy option — long system prompt, typed file and search tools, automatic context compaction. mini-SWE-agent is the light one: a single bash tool and almost no context management. OpenCode sits between them, with typed tools and a smaller preamble.
The benchmark is SWE-bench Verified, a set of real GitHub issues with tests. After removing eight tasks that need external services, 492 remained. The authors used a 447-task pool for two models and a 45-task hard subset for all five. Claude Opus 5 appears only inside Claude Code as a ceiling reference.
The important part is the control: ten pairs of runs with the same model, same harness, same tasks, and same settings. That gives an estimate of how much a score moves when nothing changes but the run. Without that baseline, a two- or three-point harness gain is unreadable.
At the task level, reruns flip 13% of outcomes at the median. Harness swaps flip the same 13%. Model swaps flip 22%. The score changes less than the flip rate because flips go both ways: some tasks that passed now fail, and some that failed now pass. But the churn is the point. A task a harness wins in one run is not reliably the task it wins in the next.
On the 447-task pool, the heaviest and lightest harnesses — Claude Code and mini-SWE-agent — are equivalent within five points for both models tested. The measured gaps were -1.8 and -1.4 points, with 90% intervals inside plus or minus five.
The one harness effect that clears the noise is a loss, not a gain. OpenCode trails by up to 9 points on the large pool. On one model, roughly half of that gap comes from runs its output cap cut short: when the model's reasoning runs long, Claude Code prompts it to continue, while OpenCode ends the step. The authors tried porting Claude Code's continuation prompt into OpenCode and could not reliably recover the gap, because the same overflow rarely triggers on a retry.
There is also a resolution problem. At the observed flip rates, 45 tasks catch a 13-point gap only about half the time. You need roughly 447 tasks to resolve a 5-point difference. A vendor's claimed 2.4-point harness lead sits below what either sample size can separate.
With the same model, same tasks, and one price list, cost per task differs by up to 3x across harnesses. The gap is set at the first call, by the fixed preamble of system prompt and tool schemas that each harness sends with every step: about 16k tokens in Claude Code, 7k in OpenCode, and under 1k in mini-SWE-agent. Per-call tool output and per-step growth differ far less. In practice, the bill is preamble size multiplied by the number of steps.
That matters for teams choosing a harness on pass rate alone. A harness can triple the cost per task while the score stays inside the noise band.
This is not an argument that harness engineering is pointless. The harness determines failure modes and cost, and the paper shows one case where a harness loses 9 points because of an output-cap interaction. Nor is it an argument that model choice is the only thing worth measuring; model swaps flip 22% of tasks, which is larger than the harness effect but still leaves plenty of variance unexplained.
The limits are real. This is a preprint, not peer reviewed. It is a single study on SWE-bench Verified with three harnesses. The 447-task pool ran only two models. The closed anchor model ran in only one harness. OpenCode's time limit on local models was not controlled. The 45-task subset cannot resolve small gaps. The authors ran only a one-model migration check on Terminal-Bench 2.1, so general-purpose agents may behave differently.
The practical reading is narrow and useful: if your harness change moves the score by less than about five points, treat it as noise until a same-configuration rerun reproduces it. If it does reproduce, check whether the gain comes from recovery, compaction, or tool reliability — the parts of the harness that stop a task from hard-failing. And before you blame the model, make sure the wrapper is not the thing setting the ceiling.
If you want a worked example of how I separate harness effects from model effects in practice, I keep a small case log here: https://hhfa008.github.io/personal/cases.html