{"slug": "harness-engineering-101-how-coding-agents-actually-work", "title": "Harness Engineering 101: How Coding Agents Actually Work", "summary": "A developer's analysis of coding agent architecture shows that swapping only the harness around a fixed model — keeping weights, tasks and context window constant — raised SWE-bench Verified bug-fixing performance from 43 to 72 tasks, citing an August arXiv paper titled \"Same Model, Different Harness.\" The writeup, which quotes Thoughtworks' Birgitta Böckeler's formulation \"Agent = Model + Harness,\" traces the field's evolution from prompt engineering to context engineering to harness engineering, and outlines an emerging outer-loop discipline. The paper's harness shortened older tool results in stages as the context window filled and prompted the agent to change approach after repeated failed commands; with a 262K window the performance gap nearly disappears.", "body_md": "Take one model and give it 169 real bug-fixing tasks from SWE-bench Verified. Keep the weights, the tasks and the context window exactly the same. Change only the agent system that runs around the model, and you will find bug-fixing task went from **43 to 72**.\n\nThat result is from a paper that went up on arXiv in August, and it is the shortest answer I have to a question I get every week: which model should we pick? That question matters less every quarter. The frontier models sit close enough together that the software wrapped around them decides most of the outcome: what a task costs, whether the agent finishes it, and whether you can trust what it hands back.\n\nThat software is the harness. Designing it is what people have started calling harness engineering.\n\nBirgitta Böckeler of Thoughtworks put it in four words, in an article on Martin Fowler's site: **Agent = Model + Harness.**\n\nThe model is the part you rent. Everything else is harness: the loop that keeps it working, the tools it can call, what goes into its context window, what it is allowed to do, and how its work gets checked before anyone accepts it. Claude Code is a harness. So is Codex CLI.\n\nThe field got here in three steps, and each one wrapped the step before it. Prompt engineering was about the words. Context engineering was about what else goes in front of the model with them: retrieved documents, memory, a summary of what happened ten turns ago. Harness engineering takes both and adds everything a model needs to act rather than talk.\n\nThe next ring is already forming. People have started calling it loop engineering: wrapping the harness in outer loops that re-run the agent on a schedule or an event, each run ending on a condition a machine can check, so nobody has to type the next instruction.\n\nThe easiest way to see a harness is to write one. This is the core of a coding agent in pseudocode. It is simplified, but every real harness I have read has this shape.\n\n``` python\ndef run_agent(task, model, tools, limits):\n    context = [system_prompt(), project_memory(), task]\n\n    for step in range(limits.max_steps):\n        if count_tokens(context) > limits.window * 0.8:\n            context = compact(context)          # summarize old turns\n\n        reply = model.generate(context, tools=tools.schemas())\n\n        if reply.is_done:\n            report = verify(reply)              # run tests, linters, a reviewer\n            if report.passed:\n                return reply\n            context.append(report.as_feedback())\n            continue\n\n        for call in reply.tool_calls:\n            if not policy.allows(call):\n                result = ask_human(call) or \"denied by policy\"\n            else:\n                result = tools.run(call)        # most of the time: a shell command\n            context.append(trim(result))        # keep the lines that matter\n\n        if same_command_failed(context, times=3):\n            context.append(\"That failed three times. Try a different approach.\")\n\n    return stop_and_report(context)\n```\n\nCount the lines that involve the model. There is one: `model.generate`. Every other line is a decision somebody had to make. When do you compact? How much of a 4,000-line test log does the model get to see? What does `policy.allows` say about `git push --force`? Change any of those answers and the same model behaves like a different agent.\n\nEvery long task eventually runs into the context limit, and what the harness does at that moment decides whether the agent finishes.\n\nThat is exactly what the August paper, \"Same Model, Different Harness\", changed. The new harness did two things. It shortened older tool results in stages as the window filled, and when it caught the agent repeating failed commands, it told it to try something else. Nothing else moved.\n\nGive the model a 262K window and the gap nearly disappears. That is also why it matters in production, where every token is billed.\n\nThe usual techniques are compaction (summarize old turns), truncation (trim old tool output, keep recent output whole), memory files loaded at the start of every session (a project's `CLAUDE.md` or `AGENTS.md`), sub-agents that take a side task into a fresh context and return only the answer, and the newest one, a full context reset.\n\nThat last one exists for a reason you would not guess. Anthropic's team building long-running apps found that \"compaction alone wasn't sufficient\". As the window filled, models started showing what they call context anxiety: wrapping the work up early because they sensed the limit coming. A clean reset with a structured handoff file worked better than a summary the model knew it was running out of room behind.\n\nAn agent that can run commands can also delete things. Every harness picks a spot on a spectrum:\n\nNone of these is wrong. The right choice depends on how much a mistake can cost, which is a question about your machine and your data rather than about the tool.\n\nThis is the layer that lets you trust the agent without reading every line it writes. Böckeler splits it in two. **Guides** steer the agent before it acts: instructions, conventions, examples. **Sensors** check the result afterwards: tests, linters, type checkers, review agents. In the pseudocode, `project_memory()` is a guide and `verify()` is a sensor.\n\nThe catch is that an agent is a poor judge of its own work. Asked to evaluate what they produced, Anthropic found agents \"tend to respond by confidently praising the work\", even when a human can see it is mediocre. Their fix was three agents: a planner writes the spec, a generator builds, and a separate evaluator tests the running app with Playwright against criteria agreed before any code was written. The solo agent took 20 minutes and $9. The full harness took six hours and $200, and the result was far better.\n\nVerification does not happen by itself. Someone has to build it, sometimes as a whole second agent whose only job is to be hard to please.\n\nMy first job was Linux server administration, and what hooked me was how much one line could do:\n\n```\ngrep \"Failed password\" /var/log/auth.log | awk '{print $(NF-3)}' | sort | uniq -c | sort -rn | head\n```\n\nEvery IP that tried to brute-force SSH on the box, counted and ranked, from five programs that know nothing about each other. Watching a coding agent work gives me the same feeling. It reaches for the same kind of tools, in roughly the order I would:\n\n``` bash\n$ rg -n \"InvoiceTotal\" src/\n$ sed -n '118,160p' src/billing/invoice.ts\n$ npm test -- invoice\n$ git diff --stat\n```\n\nGive a model one tool, a shell, and it gets every program on the machine along with it. Nobody had to build a `search_code` tool or a `run_tests` tool. `rg` and `npm test` already existed, with decades of documentation behind them.\n\nFour things make the shell fit a language model so well:\n\nThe vendors reached the same conclusion from the other side. Boris Cherny, who created Claude Code, has said early versions used RAG with a local vector database, and the team switched to plain agentic search (the model running grep and friends) because it worked better. Vercel cut an internal data agent down to little more than a single bash tool and reported it 3.5x faster on 37% fewer tokens, though on only five test queries.\n\nThe limits are worth knowing too. A shell cannot click through a web app. A SaaS product with no CLI comes in through an API or an MCP server. And the same shell that runs `npm test` can run `rm -rf`, which is why guardrails exist at all.\n\nThe harnesses I get asked about most, as of September 2026. Defaults change fast, so check the docs before relying on any cell.\n\n| Agent | Open source | Models | Default safety | Built-in tools | \n|---|---|---|---|---|\n| Claude Code | No | Claude only | Classifier on Pro, Max and Team, otherwise asks | 40+, core is Read, Edit, Grep, Glob, Bash | \n| Codex CLI | Apache 2.0 | OpenAI by default, others via config | OS sandbox, workspace only, network off | Mostly shell, plus `apply_patch` | \n| Gemini CLI | Apache 2.0 | Gemini only | No sandbox, confirms shell and writes | About 20, shell and grep among them | \n| Cursor | No | Many providers | Sandboxed shell, classifier reviews the rest | Search, read, edit, shell, browser | \n| OpenHands | MIT | Almost any, through LiteLLM | Docker sandbox in the web app, asks first in the CLI | Terminal, file editor, task tracker | \n| Aider | Apache 2.0 | Almost any, local too | No sandbox, commits each edit to git, asks before commands | No tool loop: edit formats and a repo map | \n| Cline | Apache 2.0 | Many, local too | Asks before every action | 7, with ripgrep for search | \n| Pi | MIT | 15+ providers | No sandbox, no prompts | 4: read, write, edit, bash | \n\nRead the last column top to bottom. The harnesses that lean hardest on the shell ship the fewest tools, and Codex, the most shell-centric of the big three, is also the strictest about sandboxing it. That pairing is deliberate.\n\nWhether you are picking one or building your own, measure the whole stack rather than the model alone:\n\nCost is the one people underestimate. Artificial Analysis's Coding Agent Index measured $0.07 to $2.26 per task across the model and harness pairs it tested. Most of that spread is the model, but the harness decides how many tokens the model burns on the way.\n\nAnthropic has the best line on this: \"Every component in a harness encodes an assumption about what the model can't do on its own.\" The `same_command_failed` check assumes the model will not notice it is going in circles. Compaction assumes it cannot hold a long task in one window. As models improve, some of those assumptions stop being true, and the parts built on them can go.\n\nThe part I don't expect to move is the permission boundary. A model can learn to catch its own mistakes. What it is allowed to delete on your production server stays your decision, written down in a harness, the same way it was in a sudoers file long before any of this.\n\nThe full version, including lifecycle hooks, agents beyond the code layer, and additional references, is available here: [Harness Engineering 101](https://arifulislamat.com/blog/ai/harness-engineering-101)", "url": "https://wpnews.pro/news/harness-engineering-101-how-coding-agents-actually-work", "canonical_source": "https://dev.to/arifulislamat/harness-engineering-101-how-coding-agents-actually-work-4247", "published_at": "2026-09-24 20:52:28+00:00", "updated_at": "2026-09-24 20:58:56.132413+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-research", "ai-tools", "mlops"], "entities": ["Birgitta Böckeler", "Thoughtworks", "Martin Fowler", "Claude Code", "Codex CLI", "SWE-bench Verified", "arXiv"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/harness-engineering-101-how-coding-agents-actually-work", "markdown": "https://wpnews.pro/news/harness-engineering-101-how-coding-agents-actually-work.md", "text": "https://wpnews.pro/news/harness-engineering-101-how-coding-agents-actually-work.txt", "jsonld": "https://wpnews.pro/news/harness-engineering-101-how-coding-agents-actually-work.jsonld"}}