{"slug": "coding-agent-harness-and-which-parts-of-it-actually-change-the-output", "title": "Coding agent harness, and which parts of it actually change the output", "summary": "Two September 2026 studies found that a coding agent's harness — everything except the model — changes cost far more than task success on strong models with large context windows. The UC Berkeley HarnessTax study, from a team including Ion Stoica and Matei Zaharia, ran seven models through Claude Code, Codex and Pi and found harness choice moved SWE-bench Lite success rates by no more than about 2 percent and Terminal-Bench by about 5 percent, while Claude Code cost about twice as much as Pi ($1.33 versus $0.67) for near-identical results, with Claude Fable 5 solving 97.8 percent of tasks in Claude Code and 96.7 percent in Pi. Unreal Agent's launch post claims \"up to 40% cost savings compared to Codex\" from harness design alone, though its own footnotes put savings at 16 to 28 percent against its own Codex runs.", "body_md": "# Coding agent harness, and which parts of it actually change the output\n\nA coding agent harness is everything in a coding agent except the model. Martin Fowler's site [puts it](https://martinfowler.com/articles/harness-engineering.html) as \"Agent = Model + Harness\". The harness is the loop that calls the model, the tools it can use, the files it reads at startup, how it trims its own history, whether it plans first, and what it is allowed to touch. [Claude Code](https://stackness.dev/tools/claude-code), [Codex](https://stackness.dev/tools/codex-cli), [OpenCode](https://stackness.dev/tools/opencode) and [Pi](https://stackness.dev/tools/pi) are harnesses. The model inside them is a separate choice.\n\nThe word is everywhere this month and rarely defined. Two studies from the same week of September 2026 measured which parts of a harness change the result. In short: on a strong model with a big context window, the harness changes your bill far more than your pass rate. On a weak model or a tight window, it can decide whether the task gets done at all.\n\n## **What is a harness in a coding agent stack?**\n\nMicrosoft's [definition](https://learn.microsoft.com/en-us/agent-framework/concepts/harness) is the most concrete: the harness \"drives model and tool calls, manages conversation state and context, applies approval policies, and can keep the agent progressing through a multi-step task.\" In practice that breaks into a handful of parts:\n\n- **The loop:** gather context, act, check the result, repeat.\n- **The tool set:** a few general tools like read, edit and bash, or dozens of specific ones.\n- **Startup context:** the system prompt, plus instruction files like CLAUDE.md or AGENTS.md.\n- **Context management:** what gets trimmed or summarised when the window fills up.\n- **Planning:** whether the agent writes a plan before it edits.\n- **Subagents:** separate contexts for noisy side work.\n- **Permissions and sandbox:** what the agent may run without asking.\n\nThe main harnesses have converged on most of these. Claude Code, Codex and OpenCode all read AGENTS.md, compact automatically, offer a plan mode and support subagents. Claude Code only [started reading AGENTS.md](https://simonwillison.net/2026/Sep/18/thariq-shihipar/) on 18 September, as a built-in example of \"Claude Code mods\", its upcoming way to customise the harness itself.\n\n## **Does the harness or the model change the result more?**\n\nFor pass rates on strong models, the model. For cost, the harness, by a wide margin.\n\n[HarnessTax](https://harnesstax.github.io/), from a UC Berkeley team including Ion Stoica and Matei Zaharia, ran seven models through three harnesses: Claude Code, Codex and Pi. On SWE-bench Lite the harness moved success rates by no more than about 2 percent either way, and on Terminal-Bench by about 5. Cost moved much more. Claude Code cost about twice as much as Pi on SWE-bench Lite for near-identical results. Claude Fable 5 solved 97.8 percent of tasks in Claude Code and 96.7 percent in Pi, at $1.33 against $0.67. The authors' summary: \"The same model can achieve similar success rates at up to 5x costs.\"\n\nThe likely reason is startup context. Claude Code's mean initial context is more than ten times Pi's, and Pi ships four tools: read, write, edit and bash. The study did not isolate the prompt on its own, so treat that as a strong correlation, not proof. The sample is also small, 30 tasks per benchmark, and the traces had not been released by 23 September.\n\n[Unreal Agent](https://stackness.dev/tools/unreal-agent) makes the same point as a product. Its [launch post](https://unreallabs.ai/blog/unreal-agent/) claims \"up to 40% cost savings compared to Codex\" from harness design alone: asynchronous tool calls, trimmed tool output and no subagents. The pass rates in its table are close. The footnotes matter. The 40 percent is 39, on one benchmark, against the Codex leaderboard entry. Against Unreal's own Codex runs the savings were 16 to 28 percent, with one model and a vendor running the test.\n\n## **When does context compaction actually help?**\n\nWhen the window is tight. Otherwise, barely.\n\nThe [arXiv study](https://arxiv.org/abs/2609.20804) by Run-Ze Fan and colleagues built a minimal harness and switched its parts on and off across 176 settings, on SWE-Bench Verified and Terminal-Bench, with open-weight [Nemotron](https://stackness.dev/tools/nemotron) and Mistral models. With a 32k context window, any form of context management raised SWE-Bench success by 35.7 points on average over none. At 128k the gain was 2.7 points. Without management, 78.7 percent of runs at 32k simply overflowed and died. In the authors' words: \"Context management matters most when the context-window budget is tight.\"\n\nTwo more findings. Trimming stale tool output and then summarising was the most efficient strategy. Making trimmed content recoverable added nothing, -0.36 points on average, because the models rarely asked for it back. A separate [August paper](https://arxiv.org/html/2608.26218) by Sydney Lewis found the same shape. At a 20,480-token window, trimming older tool results took one Qwen model from solving 43 tasks to 72. At 262k tokens the difference disappeared.\n\nFor most people on a frontier model with a large window, compaction is a way to keep long sessions alive, not a way to get better code. Our [compaction explainer](https://stackness.dev/blog/what-is-context-compaction-in-coding-agents-and-what-does-it-quietly-lose) covers what it loses when it does run.\n\n## **When is a planning step worth its tokens?**\n\nIt depends on the model. For the weakest model in the Fan study, Nemotron 30B, planning added 11.6 points on SWE-Bench. Without it, the model's median run collapsed from 40 turns to 5, and 68.6 percent of runs ended without a single edit. For the two strongest models, planning barely changed accuracy, at -2.0 and -0.4 points, and cut cost by about 30 percent. The saving came from dropping redundant checks after each edit, not from better code.\n\nThe authors' summary: \"Planning shifts from an accuracy scaffold for weaker models to a cost saver for stronger models.\" On a frontier model, plan mode is mostly a budget tool that also lets you review the approach before any file changes.\n\nThe same study found a similar flip for tools. The weak model did 15 percent better with predefined tools than with bash alone. The strongest did 3.6 percent better with bash alone, at 53 percent lower cost. A harness tuned for a small model can be the wrong harness for a big one.\n\n## **Which harnesses do developers run side by side?**\n\nOn Stackness, more than one at a time is common. As of 23 September 2026, 10 real public profiles list at least one coding agent harness, after removing accounts that say they are bots or test users. Four of the ten list more than one.\n\n- **Claude Code and [Cursor](https://stackness.dev/tools/cursor):** six profiles each, and three list both.\n- **Codex and OpenCode:** two profiles each. Both Codex users also list Claude Code, and one of them is the founder of Stackness.\n- **[GitHub Copilot](https://stackness.dev/tools/github-copilot) and [Windsurf](https://stackness.dev/tools/windsurf):** one profile each, and it is the same profile, which lists five harnesses.\n\nThese numbers are tiny, and we are not reading market share into them. They do match what people are building: tools for running several harnesses at once. [CC Switch](https://stackness.dev/tools/cc-switch), a desktop app that manages providers, MCP servers, skills and prompts across Claude Code, Codex, Gemini CLI, OpenCode and more, had over 135,000 GitHub stars on 23 September. [wshobson/agents](https://stackness.dev/tools/wshobson-agents), a library of agents and skills, now ships for seven harnesses from one source. [ZCode](https://stackness.dev/tools/zcode), Z.ai's harness, published its source on 20 September, after a blogger reported it uploading whole workspaces, which Z.ai said was for codebase indexing. Running several harnesses is easier when their configuration is portable, and when you can read what each one does.\n\n## **What can you change in a harness without changing models?**\n\nQuite a lot. The studies suggest where to start.\n\n- **Trim what loads at startup.** Initial context is the clearest cost driver in HarnessTax. Keep CLAUDE.md or AGENTS.md short and move long workflows into skills that load on demand, which is the move[Package domain workflows as reusable agent skills](https://stackness.dev/moves/package-domain-workflows-as-reusable-agent-skills) .\n- **Filter tool output before it lands.** A hook that cuts test output to the failures does a version of what the best context strategy in the Fan study did, before the tokens are spent.\n- **Use plan mode as a budget tool** on strong models, and as a lifeline on small ones.\n- **Change the compaction strategy** if you work in tight windows. Plugins like[fast-jev-compaction](https://stackness.dev/tools/fast-jev-compaction) prune instead of summarising.\n- **Measure your own tasks.** Every study here warns that its results belong to its own setup. Run the same task through two harnesses and compare the bill.\n\nThat last point matters most. As the Fan paper puts it, its results are \"conditional effects of the specific implementations studied rather than identifying a universally optimal harness\". The harness is now a part of your stack worth choosing on purpose, next to the model rather than after it.\n\n## **Key numbers**\n\n- **35.7 points** of extra SWE-Bench success from context management at a 32k window, against**2.7** at 128k ([Fan et al., September 2026](https://arxiv.org/abs/2609.20804) ).\n- **About 30 percent** lower cost from planning on the strongest models, for a 0.4 to 2.0 point drop in success.\n- **+11.6 points** from planning on the weakest model tested, Nemotron 30B.\n- **Within about 2 percent** is how much the harness moved success rates on SWE-bench Lite, while Claude Code cost**about twice** as much as Pi ([HarnessTax](https://harnesstax.github.io/) ).\n- **More than 10 times** Pi's initial context is what Claude Code starts with.\n- **4 of 10** real Stackness profiles with a coding agent harness list more than one, as of 23 September 2026 (see[data sources](https://stackness.dev/about/data-sources) ).\n\n## **Quick answers**\n\n**What is a coding agent harness?** Everything in a coding agent except the model: the loop, the tools, the startup instructions, context management, planning, subagents and permissions. Claude Code, Codex, OpenCode and Pi are harnesses.\n\n**Does the harness matter more than the model?** For cost, usually yes. HarnessTax found the same model scoring within about 2 percent across harnesses at up to five times the cost. For success rates on strong models, the model still matters more.\n\n**When does context compaction help?** Mostly when the context window is small. One study found a 35.7 point gain at 32k tokens and 2.7 points at 128k.\n\n**Is plan mode worth it?** On weaker models it can decide whether the task gets done. On strong models it barely changes accuracy but cut cost by about 30 percent in one study.\n\n**Why do developers run more than one harness?** Each is tuned differently, and tools like CC Switch make it cheap to move between them. On Stackness, 4 of 10 profiles with a harness list more than one.", "url": "https://wpnews.pro/news/coding-agent-harness-and-which-parts-of-it-actually-change-the-output", "canonical_source": "https://stackness.dev/blog/what-is-a-coding-agent-harness-and-which-parts-of-it-actually-change-the-output", "published_at": "2026-09-24 06:29:47+00:00", "updated_at": "2026-09-24 06:59:32.641121+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "large-language-models", "ai-research"], "entities": ["Claude Code", "Codex", "Pi", "OpenCode", "HarnessTax", "Unreal Agent", "Ion Stoica", "Matei Zaharia"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/coding-agent-harness-and-which-parts-of-it-actually-change-the-output", "markdown": "https://wpnews.pro/news/coding-agent-harness-and-which-parts-of-it-actually-change-the-output.md", "text": "https://wpnews.pro/news/coding-agent-harness-and-which-parts-of-it-actually-change-the-output.txt", "jsonld": "https://wpnews.pro/news/coding-agent-harness-and-which-parts-of-it-actually-change-the-output.jsonld"}}