# Same Coding Model, Up to 5x the Cost: The Harness Matters

> Source: <https://www.digitalapplied.com/blog/coding-agent-harness-cost-harnesstax-same-model-5x>
> Published: 2026-09-17 00:00:00+00:00

A coding agent is a model plus a harness, the program that gives the model its tools, its instructions and its loop. On September 16, 2026 researchers at UC Berkeley and Arena published HarnessTax, a study that put seven models through three harnesses, Claude Code, Codex CLI and Pi, on the same tasks. The harness changed the success rate by a few points. It changed the bill by up to five times.

This page is for anyone who pays for a coding agent by the token and has never asked whether the harness is part of the price. It defines the term, tables every one of the study's 42 measured model–harness rows with the intervals the authors give, and says what to test before switching. All figures are from the [study's page and its published chart data](https://harnesstax.github.io/), read September 18, 2026.

1. 01Success barely moved; cost moved a lot.The study reports the average harness effect on success within about ±2 points on SWE-bench Lite and ±5 on Terminal-Bench 2.0, while the same model's cost differed by up to 5x between harnesses.
2. 02Claude Code cost about twice what Pi cost.Across shared models, Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and 1.5x Pi on Terminal-Bench 2.0, by the study's geometric means of cost ratios.
3. 03A four-tool open-source harness reached the frontier.Pi, with read, write, edit and bash, sits on the cost–success frontier on both benchmarks. Its first model call carries about a tenth of Claude Code's context.
4. 04Thirty tasks per benchmark. Read the intervals.Every row is 30 tasks run three times. Most 95% intervals on success span 25 to 35 points, so small rank differences between rows are not findings.

## 01 — DefinitionWhat a harness is

When you type a request into Claude Code or Codex CLI, the model does not receive it alone. The harness wraps it in a system prompt that may run to thousands of words, attaches a list of tools the model may call, decides what parts of your repository to show it, and runs the loop: call the model, execute the tool it asked for, feed the result back, repeat until the task is done or a turn cap is hit. The study's definition is a software system that manages a model's tools, context and task execution.

Every one of those choices costs tokens. A longer system prompt is paid for on every call. A bigger tool list is a bigger schema on every call. More context per turn means more input tokens per turn. The model's price per token is fixed by its vendor; the number of tokens the harness spends to get a task done is not. That second number is what the study measures, and it is the number a team sees on its invoice. We covered the same idea from the builder's side in [our post on harness engineering and token spend](https://www.digitalapplied.com/blog/ai-harness-engineering-writer-token-spend-research-2026).

## 02 — The setupWhat the study ran

The authors, Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia, took 30 randomly sampled tasks from each of two public benchmarks. SWE-bench Lite is a set of real GitHub issues in Python repositories where the agent must produce a fix that passes hidden tests. Terminal-Bench 2.0 is a set of command-line tasks scored by each task's own checker. Each of the 21 model–harness pairs ran every task three times, in the harness's native configuration at its high-effort setting, with a cap of 100 agent turns per attempt.

The seven models were Claude Fable 5, Claude Opus 4.8, Claude Sonnet 4.6, Claude Haiku 4.5, GPT-5.6 Sol, GPT-5.6 Luna and Kimi K3, named here exactly as the study names them. The three harnesses were Claude Code from Anthropic, Codex CLI from OpenAI, and Pi, an open-source harness from earendil-works that exposes four tools. Cost is token cost at a fixed direct-API price list dated September 1, 2026, applied identically to a model in every harness. Success is the benchmark's official evaluator. Intervals are 95% bootstrap intervals over 10,000 resamples of the 30 task averages. For SWE-bench Lite the containers had no network access and the harnesses' web tools were disabled.

So it turns out that your Claude models may not need Claude Code…Melissa Z. Pan and co-authors, HarnessTax, September 16, 2026

## 03 — DatasetSWE-bench Lite: *all 21 pairs*

Success is the share of the 90 attempts (30 tasks × 3) that passed, with the 95% interval in brackets. Cost is the mean token cost per attempt in US dollars at the study's price list, with its interval. Cost per solve is the study's mean cost divided by the success rate, which is what a solved task cost on average once failed attempts are paid for. An asterisk marks a pair the study places on the cost–success frontier, meaning no other pair was both cheaper and more successful.

| Source: HarnessTax published chart data for SWE-bench Lite, 30 tasks × 3 repetitions per pair, 95% bootstrap intervals. Read September 18, 2026. * = on the study's Pareto frontier. |  |  |  | 
|---|---|---|---|
| Model · harness | Success (95% CI) | Cost / attempt (95% CI) | Cost / solve | 
|---|---|---|---|
| Claude Fable 5 · Claude Code * | 97.8% (93.3–100.0) | $1.329 (1.10–1.60) | $1.360 | 
| Claude Fable 5 · Codex | 96.7% (91.1–100.0) | $0.890 (0.70–1.13) | $0.921 | 
| Claude Fable 5 · Pi * | 96.7% (91.1–100.0) | $0.666 (0.49–0.88) | $0.689 | 
| Claude Opus 4.8 · Claude Code | 86.7% (75.6–95.6) | $0.976 (0.73–1.26) | $1.126 | 
| Claude Opus 4.8 · Codex | 88.9% (77.8–97.8) | $0.694 (0.49–0.95) | $0.780 | 
| Claude Opus 4.8 · Pi * | 82.2% (71.1–92.2) | $0.473 (0.27–0.73) | $0.575 | 
| Claude Sonnet 4.6 · Claude Code | 66.7% (51.1–81.1) | $0.669 (0.50–0.86) | $1.004 | 
| Claude Sonnet 4.6 · Codex | 68.9% (53.3–83.3) | $0.745 (0.49–1.10) | $1.081 | 
| Claude Sonnet 4.6 · Pi | 64.4% (47.8–80.0) | $0.679 (0.46–0.93) | $1.054 | 
| Claude Haiku 4.5 · Claude Code | 52.2% (36.7–67.8) | $0.426 (0.35–0.51) | $0.816 | 
| Claude Haiku 4.5 · Codex | 57.8% (41.1–74.4) | $0.392 (0.30–0.51) | $0.678 | 
| Claude Haiku 4.5 · Pi * | 60.0% (43.3–75.6) | $0.374 (0.31–0.44) | $0.623 | 
| GPT-5.6 Sol · Claude Code | 77.8% (63.3–90.0) | $1.540 (1.18–2.09) | $1.980 | 
| GPT-5.6 Sol · Codex | 73.3% (57.8–87.8) | $0.561 (0.44–0.70) | $0.765 | 
| GPT-5.6 Sol · Pi * | 74.4% (58.9–88.9) | $0.441 (0.34–0.55) | $0.592 | 
| GPT-5.6 Luna · Claude Code | 55.6% (38.9–72.2) | $0.152 (0.12–0.19) | $0.274 | 
| GPT-5.6 Luna · Codex * | 55.6% (37.8–72.2) | $0.035 (0.03–0.04) | $0.064 | 
| GPT-5.6 Luna · Pi * | 53.3% (35.6–70.0) | $0.030 (0.02–0.04) | $0.056 | 
| Kimi K3 · Claude Code | 76.7% (62.2–88.9) | $0.784 (0.57–1.04) | $1.023 | 
| Kimi K3 · Codex | 74.4% (60.0–87.8) | $0.845 (0.67–1.04) | $1.135 | 
| Kimi K3 · Pi | 72.2% (57.8–85.6) | $0.455 (0.32–0.61) | $0.630 | 

The headline pair is the top row. Claude Fable 5 solved 97.8% of attempts in Claude Code and 96.7% in Pi, with intervals that overlap almost entirely, at $1.33 against $0.67 per attempt. The widest gap in the data, which matches the study's "up to 5x", is GPT-5.6 Luna: $0.15 per attempt in Claude Code against $0.03 in Pi, for 55.6% and 53.3% success. Note also the last row: GPT-5.6 Sol in Claude Code was the most expensive pair on this benchmark at $1.54, more than the strongest model in the same harness.

#### Mean cost per attempt on SWE-bench Lite, all 21 pairs

HarnessTax chart data, UC Berkeley and Arena, September 16, 2026. Token cost at a fixed price list dated September 1, 2026.
## 04 — DatasetTerminal-Bench 2.0: all 21 pairs

Same columns, same reading. The harness effect on success is wider here, about ±5 points on the study's average, and the order of harnesses changes for some models. Haiku 4.5 in Claude Code hit the 100-turn cap on four attempts, all unsolved, which the study's data flags on that row.

| Source: HarnessTax published chart data for Terminal-Bench 2.0, 30 tasks × 3 repetitions per pair, 95% bootstrap intervals. Read September 18, 2026. * = on the study's Pareto frontier. |  |  |  | 
|---|---|---|---|
| Model · harness | Success (95% CI) | Cost / attempt (95% CI) | Cost / solve | 
|---|---|---|---|
| Claude Fable 5 · Claude Code | 75.6% (61.1–88.9) | $1.554 (0.99–2.24) | $2.057 | 
| Claude Fable 5 · Codex | 72.2% (56.7–86.7) | $0.976 (0.63–1.41) | $1.351 | 
| Claude Fable 5 · Pi | 71.1% (54.4–85.6) | $1.079 (0.57–1.75) | $1.517 | 
| Claude Opus 4.8 · Claude Code | 68.9% (52.2–84.4) | $0.899 (0.59–1.28) | $1.305 | 
| Claude Opus 4.8 · Codex | 72.2% (56.7–86.7) | $0.848 (0.51–1.27) | $1.174 | 
| Claude Opus 4.8 · Pi | 72.2% (58.9–84.4) | $0.758 (0.38–1.25) | $1.049 | 
| Claude Sonnet 4.6 · Claude Code | 62.2% (46.7–76.7) | $0.669 (0.45–0.92) | $1.075 | 
| Claude Sonnet 4.6 · Codex | 63.3% (46.7–78.9) | $0.552 (0.38–0.74) | $0.872 | 
| Claude Sonnet 4.6 · Pi | 65.6% (48.9–82.2) | $0.614 (0.39–0.87) | $0.937 | 
| Claude Haiku 4.5 · Claude Code | 41.1% (27.8–55.6) | $0.263 (0.16–0.38) | $0.639 | 
| Claude Haiku 4.5 · Codex | 31.1% (15.6–47.8) | $0.214 (0.14–0.29) | $0.687 | 
| Claude Haiku 4.5 · Pi | 47.8% (32.2–63.3) | $0.250 (0.16–0.36) | $0.522 | 
| GPT-5.6 Sol · Claude Code | 71.1% (55.6–85.6) | $1.355 (0.95–1.85) | $1.905 | 
| GPT-5.6 Sol · Codex | 78.9% (65.6–91.1) | $0.761 (0.46–1.13) | $0.965 | 
| GPT-5.6 Sol · Pi * | 83.3% (71.1–94.4) | $0.421 (0.27–0.61) | $0.505 | 
| GPT-5.6 Luna · Claude Code | 70.0% (55.6–83.3) | $0.098 (0.07–0.13) | $0.141 | 
| GPT-5.6 Luna · Codex | 72.2% (57.8–85.6) | $0.064 (0.04–0.09) | $0.089 | 
| GPT-5.6 Luna · Pi * | 76.7% (62.2–90.0) | $0.045 (0.03–0.07) | $0.059 | 
| Kimi K3 · Claude Code | 66.7% (52.2–80.0) | $0.521 (0.36–0.71) | $0.782 | 
| Kimi K3 · Codex | 70.0% (55.6–83.3) | $0.450 (0.28–0.66) | $0.643 | 
| Kimi K3 · Pi | 73.3% (60.0–85.6) | $0.383 (0.24–0.54) | $0.523 | 

Two rows the study singles out. GPT-5.6 Sol reached 83.3% in Pi against 78.9% in Codex, its vendor's own harness, at about half the cost, $0.42 against $0.76. And the only two frontier pairs on this benchmark are both Pi. Across the six Anthropic and OpenAI models and both benchmarks, the study counts nine of twelve comparisons where a harness other than the model vendor's own gave the highest observed success rate. The authors' reading is that a model's capabilities carry over to other harnesses; ours is that the vendor's harness is a default, not a measurement.

## 05 — The mechanismWhere the cost goes

The study looked at turns and at initial context to explain the gap. For Claude Fable 5 on SWE-bench Lite, Pi and Claude Code averaged 15.4 and 15.3 turns per attempt, almost identical, yet Claude Code cost about twice as much for a 1.1-point gain in success. The money is not in extra turns; it is in what each turn carries. Across all seven models, Claude Code's mean context on the first model call was over ten times Pi's, with longer instructions and larger tool schemas. The authors note that total spend also depends on caching, generated tokens and later calls, so the first-call figure is a cause, not the whole account.

The study calls the difference a "harness tax", borrowing the phrase from an April 2026 Portkey blog post, and says a user pays it whenever they accept a coding agent's default harness without comparing alternatives. The point is not that a richer harness is wrong. The authors say richer features may still help other models, workloads or interactive sessions, and that harness complexity should be treated as an empirical trade-off. On these two benchmarks, with these seven models, the extra context did not buy extra solved tasks.

Token cost at API list prices is not what most people pay. A Claude Code or Codex subscription flattens the per-token bill into a monthly fee with usage limits, and a harness that spends more tokens costs a subscriber nothing extra until it hits the limit. The study's numbers apply directly to API billing and to any harness pointed at a pay-per-token key. For a subscriber the relevant unit is tasks per week before the limit, which the same data predicts but does not state.

## 06 — DecisionWhat to do with this

Do not switch harnesses on the strength of this page. Do measure. The study's own last section says the next step is evaluating harnesses in real development workflows, where tasks span sessions and developers give feedback, and that is exactly what a team can do for itself in a week.

1. **Pick ten recent tickets your agent already closed.** Real ones from your repository, not benchmark tasks. Include two the agent failed.
2. **Run each in two harnesses with the same model and an API key.** Same effort setting, same turn cap. Record whether the pull request would have merged and the token cost from the provider's dashboard.
3. **Compute cost per merged PR, not cost per run.** A cheaper harness that fails more often is not cheaper. The study's cost-per-solve column is the same arithmetic.
4. **Check what you would lose.** Skills, hooks, memory files and permission rules are harness features, and they do not all travel. Our post on[cross-harness skill portability](https://www.digitalapplied.com/blog/codex-cli-cross-harness-skill-portability-lock-in) lists what moves between Claude Code and Codex CLI and what does not.
5. **Re-run when the model changes.** The study's rankings are for these seven models at September 2026 prices. As we argued in[models swap in 48 hours, the harness does not](https://www.digitalapplied.com/blog/models-swap-in-48-hours-the-harness-does-not) , the harness is the part of the stack a team keeps longest, so its cost profile deserves a measurement of its own. If you want that measurement run against your own repository, our[AI transformation service](https://www.digitalapplied.com/services/ai-transformation) includes it.

## 07 — How to read thisMethodology

Every figure on this page is from one study. We copied the numbers from the study's published chart data files rather than from screenshots, and we ran no experiments ourselves.

- Who ran it
- HarnessTax, by Melissa Z. Pan, Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia of UC Berkeley and Arena, published September 16, 2026. The acknowledgements list API credits from Arena, Anthropic API credits from Laude Institute, and AWS credits from an Amazon fellowship.
- Sample size
- 30 randomly sampled tasks per benchmark, each run three times per model–harness pair, so 90 attempts per row and 21 rows per benchmark. Attempts were capped at 100 agent turns.
- Cost basis
- Token cost at a fixed direct-API price list dated September 1, 2026, applied to the same model identically across harnesses. Subscription pricing is not modelled. Kimi K3 was accessed through Fireworks AI in its single native thinking mode.
- Intervals
- 95% confidence intervals from 10,000 bootstrap resamples of the 30 task averages, as the study describes. We print them beside every success and cost figure; the study's frontier marking is copied as given.
- As-of date
- The study page and its chart data were read on September 18, 2026. This page is dated to the editorial day after publication; the collection date is stated here and in the dataset card only.
- Known limitations
- The authors state that both benchmarks are public and the models may have seen them in training, that results may differ on other workloads, and that turn definitions differ between harnesses. Thirty tasks give wide intervals; a difference inside them is not a finding. The study says it will release its profiling traces; as of our read they were not yet linked.

## 08 — Next stepThe harness is a line on the invoice

### Run ten of your own tickets through a second harness this week

The study's numbers are for its tasks, its price list and its September 2026 models. Yours will differ, and the only way to know by how much is to hold the model fixed and change the harness on work you already did. Record cost per merged pull request, keep the harness whose number is lower, and re-run the test when you next change models.