{"slug": "harness-engineering-the-engineering-discipline-behind-reliable-ai-agents", "title": "Harness Engineering: The Engineering Discipline Behind Reliable AI Agents", "summary": "Anthropic's engineering team is advancing the discipline of harness engineering, which focuses on the systems surrounding AI models—context, tools, execution environments, and feedback loops—that enable reliable agentic behavior. The team argues that as models become more capable, the harness does not disappear but evolves to handle new failure modes from more ambitious tasks, making the harness a key determinant of system reliability.", "body_md": "For years, the central question in AI engineering was:\n\nWhich model should we use?\n\nGPT vs Claude.\n\nReasoning models vs general-purpose models.\n\nBigger context vs smaller context.\n\nOne model vs another.\n\nBut as AI systems evolve from chatbots into agents that can inspect repositories, modify code, run commands, browse the web, interact with APIs, execute tests and work for hours without constant supervision, the question is changing.\n\nThe interesting problem is no longer simply how capable the model is.\n\nIt is:\n\nHow do we build a system that allows a probabilistic model to do reliable engineering work?\n\nThat is the problem space increasingly described as **Harness Engineering**.\n\nThe term has emerged as a way of thinking about everything surrounding the model that turns raw intelligence into an operational agent: context, tools, execution environments, permissions, state, orchestration, memory, feedback, verification and recovery.\n\nA useful mental model is:\n\nAgent = Model + Harness\n\nThe model provides the intelligence.\n\nThe harness determines how that intelligence is exposed to the world, constrained, observed, corrected and ultimately turned into useful work.\n\nAnd that distinction is becoming increasingly important.\n\nA language model, by itself, is remarkably limited.\n\nIt receives information and generates a response.\n\nIt doesn't inherently have:\n\nThose capabilities come from the system around the model.\n\nA coding agent might look deceptively simple from the outside:\n\n```\nUser\n ↓\nModel\n ↓\nCode\n```\n\nBut the actual system looks more like:\n\n```\n                 ┌─────────────────┐\n                 │   Human Intent  │\n                 └────────┬────────┘\n                          ↓\n                 ┌─────────────────┐\n                 │ Context & Rules │\n                 └────────┬────────┘\n                          ↓\n                 ┌─────────────────┐\n                 │      Model      │\n                 └────────┬────────┘\n                          ↓\n                 ┌─────────────────┐\n                 │ Tools & Runtime │\n                 └────────┬────────┘\n                          ↓\n                 ┌─────────────────┐\n                 │ Feedback / Evals│\n                 └────────┬────────┘\n                          ↓\n                 ┌─────────────────┐\n                 │ Recovery / Loop │\n                 └────────┬────────┘\n                          │\n                          └──────→ Continue\n```\n\nThat entire surrounding system is the harness.\n\nThe Harness Engineering material makes this distinction explicitly: the harness includes prompts, tools, skills, infrastructure and orchestration—the parts that make the model useful as an agent.\n\nThis leads to a surprisingly important conclusion:\n\nThe model may be the most intelligent component in the system, but it isn't necessarily the component that determines system reliability.\n\nIt is tempting to believe that sufficiently capable models will eventually eliminate the need for all this scaffolding.\n\nIf the model can reason well enough, why do we need elaborate instructions, verification loops, context management and recovery mechanisms?\n\nThere is some truth to this.\n\nAs models improve, certain pieces of the harness become unnecessary. Anthropic has explicitly described cases where behaviors previously compensated for by the harness improved in newer models, allowing parts of the harness to be removed.\n\nBut this creates a more interesting dynamic.\n\nThe harness doesn't disappear.\n\n**The operating envelope expands.**\n\nA better model allows us to attempt harder tasks.\n\nHarder tasks introduce new failure modes.\n\nThe harness evolves to handle those failures.\n\nSo the progression looks less like:\n\n```\nBetter model → Less harness\n```\n\nand more like:\n\n```\nBetter model\n     ↓\nMore capable agent\n     ↓\nMore ambitious tasks\n     ↓\nNew failure modes\n     ↓\nBetter harness\n     ↓\nEven more capable agent\n```\n\nAnthropic's recent harness work makes a similar point: every harness contains assumptions about what a model cannot reliably do, and those assumptions need to be revisited as models improve.\n\nThat is why Harness Engineering isn't simply a collection of temporary workarounds.\n\nIt is becoming an ongoing engineering discipline.\n\nThere is an interesting evolution here.\n\nInitially, the focus was:\n\nWhat should I tell the model?\n\nWe experimented with wording, examples, roles, instructions and few-shot prompts.\n\nThen agents became longer-running and more capable.\n\nThe question became:\n\nWhat information should the model have at this moment?\n\nThat is **Context Engineering**.\n\nAnthropic describes context engineering as the broader problem of curating the information available to a model during inference—including system instructions, tools, MCP servers, external data and conversation history.\n\nAnd then comes the next question:\n\nWhat environment should the model operate in?\n\nThat is where **Harness Engineering** begins.\n\nNow we're thinking about:\n\nThe progression is therefore:\n\nPrompt → Context → Harness\n\nEach step expands the engineering boundary.\n\nOne of the biggest mistakes in agent design is assuming:\n\nMore context = better results.\n\nIt doesn't.\n\nA model may technically be able to accept a huge amount of information while becoming less effective at using it.\n\nAnthropic describes context as a finite attention resource and recommends optimizing for the smallest set of high-signal information that maximizes the desired outcome.\n\nThis changes how we think about agent memory.\n\nA mature agent shouldn't simply receive:\n\n```\nEverything we know about the project\n```\n\nIt should receive:\n\n```\nWhat matters for this task\n+ what the agent needs to find\n+ what it needs to remember\n```\n\nThat leads to several important harness primitives:\n\nThe Harness Engineering analysis makes a similar distinction between long-lived instructions, persistent memory, session continuity and temporary conversation, emphasizing that context should be governed rather than endlessly accumulated.\n\nThis is why context engineering is not simply prompt writing.\n\nIt is **information architecture for an intelligent runtime**.\n\nThis is one of the most interesting consequences of agent-first development.\n\nHumans can rely on knowledge that exists outside the repository:\n\nAn agent can't.\n\nIf the relevant knowledge isn't accessible during execution, it effectively doesn't exist for the agent.\n\nOpenAI's experience building an internal product almost entirely through Codex pushed this idea particularly far. They treated repository knowledge as the system of record and deliberately worked to make the codebase itself legible to agents.\n\nThis means the repository increasingly contains:\n\nThe repository is no longer just where the code lives.\n\nIt becomes part of the **agent's operating environment**.\n\nAnd this gives us a new architectural quality:\n\nA system should not only be understandable to humans.\n\nIt should be **discoverable and understandable by agents**.\n\nThat means good architecture increasingly has two audiences:\n\nHuman developers and machine collaborators.\n\nA model that generates text can make a bad suggestion.\n\nAn agent with tools can make a bad change.\n\nThat is a fundamental difference.\n\nOnce an agent can:\n\n```\nRead\n ↓\nModify\n ↓\nExecute\n ↓\nObserve\n ↓\nModify again\n```\n\nwe've moved from generation to execution.\n\nThis is why tools aren't merely features of an agent.\n\nThey are **execution interfaces**.\n\nThe harness needs to determine:\n\nThe Harness Engineering material describes tools in exactly this way: managed execution interfaces that need authorization, scheduling, interruption and proper closure, with higher-risk tools receiving stronger governance.\n\nA filesystem read and arbitrary shell execution may both be “tools,” but treating them identically would be poor engineering.\n\nThe principle is simple:\n\nCapability should always be accompanied by control.\n\nTraditional applications often treat authorization as an infrastructure concern.\n\nFor autonomous agents, permissions directly shape behavior.\n\nConsider:\n\n```\nReadFile\n```\n\nversus:\n\n```\nBash\n```\n\nThe first is primarily observational.\n\nThe second can potentially:\n\nSo a mature harness needs something more sophisticated than:\n\n```\nTool → Execute\n```\n\nIt needs:\n\n```\nAgent requests action\n        ↓\nPolicy evaluation\n        ↓\n   Allow / Deny / Ask\n        ↓\n     Execute\n        ↓\n    Observe\n        ↓\n    Record state\n```\n\nThe important insight is that the permission layer isn't merely protecting the user.\n\nIt is also protecting the **agent runtime from its own mistakes**.\n\nThe Harness Engineering material captures this philosophy well: permission is an organ of the system rather than an accessory feature.\n\nA chatbot can be modeled as:\n\n```\nInput → Model → Output\n```\n\nA real agent needs a loop.\n\nSomething closer to:\n\n```\nInput\n ↓\nPrepare state\n ↓\nInvoke model\n ↓\nConsume response\n ↓\nExecute tools\n ↓\nObserve results\n ↓\nUpdate state\n ↓\nRecover if necessary\n ↓\nContinue\n ↓\nStop when appropriate\n```\n\nThe Harness Engineering analysis calls this the **query loop—the heartbeat of the agent system**.\n\nThe model invocation is therefore only one part of the runtime.\n\nThe loop owns:\n\nThis is an important maturity test.\n\nA system that can answer a prompt is a demo.\n\nA system that can maintain coherent progress through changing state, failures and tool calls is an **agent runtime**.\n\nTraditional software architecture often starts with:\n\nHappy path\n\nand then adds:\n\nError handling.\n\nAgent systems need a different mindset.\n\nFor long-running agents:\n\nFailure is part of the main path.\n\nContext limits will be reached.\n\nTools will fail.\n\nCommands will return unexpected output.\n\nNetworks will break.\n\nThe model will misunderstand something.\n\nA subagent may fail.\n\nA hook may loop.\n\nA recovery mechanism may itself fail.\n\nThe Harness Engineering principles explicitly argue that error paths should be designed as first-class paths, with retry limits, circuit breakers and anti-loop mechanisms established at design time.\n\nThe goal isn't to build an agent that never fails.\n\nThat's unrealistic.\n\nThe goal is to build one that can **fail without losing control of the workflow**.\n\nImagine an agent has spent an hour:\n\nThen its context becomes unusable.\n\nA naïve system says:\n\nStart again.\n\nA good harness asks:\n\nWhat useful state can we preserve?\n\nThis is where durable artifacts, Git, plans, memory and structured state become extremely valuable.\n\nAnthropic's work on long-running agents emphasizes the use of structured artifacts and task decomposition to allow work to continue coherently across sessions.\n\nRecovery isn't simply:\n\n“Try again.”\n\nIt is:\n\n“Continue from the best known state.”\n\nThat distinction becomes increasingly important as agent tasks move from minutes to hours and eventually days.\n\nPerhaps the most dangerous sentence in agentic development is:\n\n“The agent says it's done.”\n\nImplementation and correctness are different things.\n\nAn agent can:\n\nThis is why verification needs to be an explicit stage.\n\nAnthropic's recent long-running coding work uses a planner, generator and evaluator architecture, with the evaluator independently examining the resulting application against concrete criteria.\n\nThe Harness Engineering analysis similarly emphasizes that verification should be independent rather than simply being the implementer's final self-check.\n\nA useful pattern is:\n\n```\nResearch\n   ↓\nImplementation\n   ↓\nVerification\n   ↓\nFailure?\n ┌─┴─┐\nYes  No\n ↓    ↓\nFix   Done\n ↓\nVerify again\n```\n\nThe important thing isn't necessarily having multiple agents.\n\nIt is **separating responsibilities**.\n\nThere's a tendency to think:\n\nMore agents = more intelligence.\n\nThat's not necessarily true.\n\nFive agents performing the same task can simply produce five different versions of the same uncertainty.\n\nThe more interesting design is:\n\n```\nResearch\n   ↓\nSynthesis\n   ↓\nImplementation\n   ↓\nVerification\n```\n\nEach stage has a different responsibility.\n\nThe Harness Engineering material describes the value of multi-agent systems in exactly these terms: research, implementation, verification and synthesis become separate responsibility containers, with the coordinator responsible for reconverging distributed knowledge.\n\nThis produces an important principle:\n\nParallelism is useful when it creates better boundaries, not merely more output.\n\nAnd synthesis becomes critical.\n\nResearch can be distributed.\n\n**Understanding must reconverge.**\n\nMartin Fowler's treatment of Harness Engineering introduces another useful distinction: **feedforward and feedback**.\n\nTell the agent what good behavior looks like before it acts.\n\nExamples:\n\nTell the agent what happened after it acted.\n\nExamples:\n\nA system relying only on feedforward assumes the instructions are sufficient.\n\nA system relying only on feedback forces the agent to repeatedly discover the same mistakes.\n\nA strong harness combines both:\n\n```\n                INTENT\n                  ↓\n             Instructions\n                  ↓\n               Agent\n                  ↓\n              Change\n                  ↓\n        ┌─────────┴─────────┐\n        ↓                   ↓\n Computational          Inferential\n feedback               feedback\n        ↓                   ↓\n tests / types          review / evals\n        └─────────┬─────────┘\n                  ↓\n                Fix\n                  ↓\n                Done\n```\n\nThis is essentially a **feedback-controlled engineering system**.\n\nOne of the most powerful ideas in agent-first engineering is:\n\nDon't just document architectural rules. Make them executable.\n\nHumans can remember:\n\n“This module shouldn't depend on that module.”\n\nAgents need stronger signals.\n\nOpenAI's agent-first engineering experiment used explicit architectural constraints and custom tooling to make structural rules enforceable rather than relying purely on documentation.\n\nThis changes how we think about architecture.\n\nA good architecture for agentic development isn't simply:\n\nunderstandable.\n\nIt is:\n\ndiscoverable + testable + enforceable.\n\nThe same applies to:\n\nAnything important enough to repeatedly explain may be important enough to **encode mechanically**.\n\nSome codebases are simply easier for agents to work with than others.\n\nA codebase with:\n\nprovides many more useful signals to an agent.\n\nA chaotic codebase gives the model far more ambiguity.\n\nMartin Fowler calls attention to these environmental properties as **ambient affordances**—characteristics of the environment that make it easier for an agent to navigate and operate effectively.\n\nThis suggests that we may eventually evaluate architectures partly by a new criterion:\n\nHow harnessable is this system?\n\nNot just:\n\nBut also:\n\nCan an agent reliably understand, modify and verify it?\n\nThis doesn't mean engineers disappear.\n\nIt changes where engineering effort is applied.\n\nOpenAI's Codex experiment provides a striking example. With humans intentionally not writing application code, the engineers increasingly focused on defining intent, creating scaffolding, improving feedback loops and making the environment more legible to agents.\n\nTheir phrase captures the shift:\n\nHumans steer. Agents execute.\n\nThe engineer increasingly becomes responsible for:\n\nThe agent increasingly handles:\n\nThis produces a powerful distinction.\n\nSuppose an agent repeatedly makes the same mistake.\n\nYou can:\n\nTell the agent again.\n\nBuild a mechanism that prevents the mistake.\n\nMaybe that's:\n\nOption A fixes one interaction.\n\nOption B improves **every future interaction**.\n\nThat is engineering leverage.\n\nThere is another consequence that becomes visible once agents can produce code much faster than humans can review it.\n\nTraditional engineering is constrained by human throughput.\n\nAgents change that.\n\nOpenAI's internal experiment reported roughly 1,500 pull requests merged over five months while the team scaled from three engineers to seven, with humans increasingly operating at a higher level of abstraction.\n\nAt that scale, some traditional processes become bottlenecks.\n\nIf an agent can produce ten changes while a human can carefully review one, simply adding more human review doesn't necessarily solve the problem.\n\nThe solution becomes:\n\nAutomate more of the feedback.\n\nThat means:\n\nThis changes the economics of quality.\n\nWhen generation becomes extremely cheap, **verification becomes one of the scarce resources**.\n\nHigh throughput creates another challenge:\n\n**software entropy.**\n\nIf agents can generate thousands of changes, they can also generate thousands of mediocre decisions.\n\nOver time you may accumulate:\n\nOpenAI describes addressing this with recurring “garbage collection” workflows in which agents inspect the repository for violations of established principles and clean them up.\n\nThis leads to an important principle:\n\nWhen code generation becomes cheap, code cleanup must become continuous.\n\nAgentic engineering therefore needs not only generation loops but **maintenance loops**.\n\nThe coding loop looks like:\n\n```\nPrompt\n ↓\nCode\n ↓\nTest\n ↓\nFix\n```\n\nHarness Engineering introduces a higher-level loop:\n\n```\nAgent fails\n    ↓\nUnderstand why\n    ↓\nIdentify missing capability\n    ↓\nImprove harness\n    ↓\nAgent retries\n    ↓\nFailure becomes less likely\n```\n\nThis is a major shift in mindset.\n\nInstead of constantly improving **outputs**, we improve the **system producing the outputs**.\n\nOpenAI's experience illustrates this philosophy clearly: when an agent struggled, engineers focused on identifying the missing capability and making it both legible and enforceable rather than simply asking the model to try harder.\n\nThat may be the defining idea of Harness Engineering.\n\nDon't just fix the failure. Improve the environment that allowed the failure to happen.\n\nThis is where the idea becomes particularly interesting.\n\nIf:\n\n```\nAgent = Model + Harness\n```\n\nthen we can ask:\n\nCan the harness itself be optimized by an agent?\n\nThe supplied Harness Evolution research proposes exactly this.\n\nA worker agent performs a task.\n\nAn evaluator examines the result.\n\nAn evolution agent studies the failures and modifies the harness.\n\nThen the worker runs again.\n\nConceptually:\n\n```\n             WORKER\n                ↓\n             OUTPUT\n                ↓\n           EVALUATOR\n                ↓\n         Failure analysis\n                ↓\n         EVOLUTION AGENT\n                ↓\n          Better Harness\n                ↓\n             WORKER\n                ↺\n```\n\nAnd then comes an even more ambitious concept:\n\nIf the harness can evolve, why not optimize the **process used to evolve harnesses**?\n\nThe research describes an outer loop that evaluates harness-evolution strategies across multiple tasks and attempts to improve the evolution blueprint itself.\n\nIn other words:\n\n```\nModel\n  ↓\nAgent\n  ↓\nHarness\n  ↓\nHarness Evolution\n  ↓\nMeta-Harness Evolution\n```\n\nWe move from:\n\nengineering the agent\n\nto:\n\nengineering the system that improves agents.\n\nThat could become one of the most interesting areas of AI engineering over the next few years.\n\nIf you're building an AI agent today, don't begin with:\n\n“Which model should I use?”\n\nStart with these questions.\n\nAnd finally:\n\nWhen the agent repeatedly fails, how does the system itself get better?\n\nThat last question is arguably the most important one.\n\nThe deeper implication of Harness Engineering is that the unit of engineering is changing.\n\nTraditional software engineering largely asked:\n\nHow do humans build software?\n\nAI-assisted development asked:\n\nHow can humans use AI to build software faster?\n\nAgent-first engineering asks something different:\n\nHow do we design an environment in which AI can reliably build software?\n\nThat environment includes:\n\n```\n             HUMAN INTENT\n                   ↓\n          ┌─────────────────┐\n          │     HARNESS     │\n          │                 │\n          │ Context         │\n          │ Tools           │\n          │ Permissions     │\n          │ State           │\n          │ Memory          │\n          │ Orchestration   │\n          │ Verification    │\n          │ Recovery        │\n          │ Governance      │\n          └────────┬────────┘\n                   ↓\n                MODEL\n                   ↓\n                ACTION\n                   ↓\n               FEEDBACK\n                   ↓\n              CORRECTION\n                   │\n                   └──────────→ CONTINUE\n```\n\nThe model remains the source of intelligence.\n\nBut the harness increasingly determines **how that intelligence behaves in the world**.\n\nThe first wave of generative AI taught us how to **prompt models**.\n\nThe next wave taught us how to **engineer context**.\n\nThe emerging wave is teaching us how to **engineer the systems around models**.\n\nThat's Harness Engineering.\n\nIt isn't simply a better system prompt.\n\nIt isn't adding more tools.\n\nIt isn't throwing more agents at a problem.\n\nIt is a recognition of a fundamental engineering reality:\n\nA probabilistic component cannot simply be placed inside a deterministic workflow and expected to inherit its reliability.\n\nReliability has to be engineered around it.\n\nWe need context that is curated rather than dumped.\n\nTools that are powerful but governed.\n\nState that survives interruptions.\n\nFeedback that catches mistakes.\n\nVerification that doesn't simply rubber-stamp implementation.\n\nRecovery that preserves momentum.\n\nArchitecture that is legible and enforceable.\n\nAnd organizational knowledge that is encoded into the environment rather than trapped inside individual people's heads.\n\nThe most capable agent of the future may therefore not simply be the one with the smartest model.\n\nIt may be the one with the **best environment in which that model can operate**.\n\nAnd perhaps that is the most important shift in perspective:\n\nWe are no longer just engineering software with AI.\n\nWe are engineering the system in which AI engineers software.\n\n**That is Harness Engineering.**", "url": "https://wpnews.pro/news/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents", "canonical_source": "https://dev.to/vnayak_hejib/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents-kap", "published_at": "2026-09-03 14:25:21+00:00", "updated_at": "2026-09-03 14:55:45.041872+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "ai-research"], "entities": ["Anthropic"], "alternates": {"html": "https://wpnews.pro/news/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents", "markdown": "https://wpnews.pro/news/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents.md", "text": "https://wpnews.pro/news/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents.txt", "jsonld": "https://wpnews.pro/news/harness-engineering-the-engineering-discipline-behind-reliable-ai-agents.jsonld"}}