Your model may not be the thing that’s failing #
An agent's reliability hinges not just on its underlying large language model, but critically on its harness: the surrounding architecture of rules, tools, workflows, memory, and guardrails. Focusing solely on model capability often overlooks the operational framework that truly dictates performance and prevents mistakes in production.
A significant shift is underway in how we construct these harnesses. The "old way" involved hand-coding intricate orchestration logic in Python or TypeScript. Today, much of an agent's behavior is defined and managed declaratively using flexible, editable Markdown files for instructions, skills, and configuration. This parallels the robust, self-improving systems seen in advanced AI coding agents.
This declarative approach makes harness engineering a practical, iterative cycle. Instead of hoping a stronger model will compensate for systemic flaws, developers inspect agent failures in production traces, identify missing instructions or flawed processes, and then directly improve the harness. This means updating a Markdown instruction file or refining a skill definition, rather than rewriting core code.
This method also draws a direct parallel between coding agent components and production business agents. Rules and instruction files map to system prompts, subagents become multi-agent orchestrations, and skills function as structured standard operating procedures. This 1:1 mapping streamlines the development and continuous refinement of robust, adaptable AI agents.
The coding-agent playbook now fits business bots #
The architectural blueprints for reliable AI coding agents now directly inform the design of production-grade business bots. This shift redefines how we build and manage agentic systems, moving away from brittle, hardcoded logic towards flexible, declarative configurations.
Consider the direct mapping: instruction files, like INSTRUCTIONS.md, become system rules that define an agent's global context and persona. Subagents, previously used for task decomposition in coding, now represent delegated workflows in multi-agent orchestration.
Crucially, MCP tools (Model Context Protocol) provide standardized integration points for external databases and APIs, while skills—defined as markdown recipes—replace complex functions with structured, step-by-step procedural guidelines. Finally, hooks enforce guardrails, acting as policy interceptors for pre- and post-execution checks, ensuring deterministic control.
This declarative approach, leveraging markdown and YAML files, makes workflows inherently easier to inspect and revise. Agents can even self-modify these configurations based on runtime feedback, a core tenet of harness engineering. Code still handles essential plumbing and deterministic controls, but the high-level logic resides in human- and AI-readable configuration.
While Cole Medin’s video uses LangChain’s Managed Deep Agents as an implementation example, this underlying architecture is universal. The same principles of externalizing rules, skills, and guardrails can be applied with other deployment and orchestration platforms, emphasizing the strategic importance of the design pattern over any specific tool.
Two loops turn mistakes into operational memory #
Mistakes are not failures, but data points. The agentic system captures these data points through two distinct feedback loops, transforming runtime errors into operational memory.
First, the runtime memory loop addresses immediate uncertainties. When an agent encounters a novel or ambiguous situation, it can offer a cautious, provisional response, then escalate the case to a human for review. The human confirms or corrects the answer, and this correction is immediately written back into the agent’s memory.
This correction becomes a retrievable lesson, stored in a context hub. This memory can be scoped to an individual user, ensuring personalized learning, or shared across relevant cases for broader system improvement. The agent learns not just what to do, but how to adapt its internal rules and workflows.
Consider a support agent for Archon. A user reported a local fix missing, but the agent, referencing an origin/main worktree, couldn't find it. After human correction, the agent recorded the specific context—the user’s branch and the correct local fix location. A subsequent, related query could then be answered confidently, drawing directly from this newly acquired, specific memory. For further reading on robust agent design, see Building Effective AI Agents - Anthropic Engineering.
The second loop, the retroactive trace loop, scrutinizes an agent's entire interaction history, a rich "gold mine" of performance data. This loop identifies deeper, systemic issues that the runtime loop might miss.
Enjoying this? Get one like it in your inbox each morning.
one email a day · unsubscribe in two clicks · no third-party tracking
Mine the traces—but keep humans on the brakes #
A slower, retroactive loop mines production traces for recurring failures. This "gold mine" of information allows an agent to analyze its own operational history and propose updates to its core instructions, skills, or guardrails, moving beyond single-case corrections.
Crucially, distinguish useful memory from a durable harness fix. Runtime memories retrieve case-specific lessons for immediate application, like a customer's prior preference. Workflow and instruction updates, however, prevent repeat failures more broadly, hardening the agent's underlying logic.
Implementing these changes requires caution and a human on the brakes. Inspect proposed edits from the agent, then test them against a diverse set of representative cases. For consequential checks, implement deterministic hooks to enforce policy. Finally, deploy updates incrementally and monitor traces after rollout to confirm the fixes and prevent new regressions in the harness engineering.
Frequently Asked Questions #
What is harness engineering for AI agents?
Harness engineering improves the rules, workflows, tools, memory, and guardrails surrounding an AI model so its behavior becomes more reliable.
How can an AI agent learn from production mistakes?
Capture uncertain or failed interactions, review them with a human, and turn confirmed corrections into retrievable memories or updates to the agent’s instructions and skills.
Should an AI agent rewrite its own instructions automatically?
It can propose changes, but consequential edits should be reviewed and tested by a human before deployment.
What role do production traces play?
Traces reveal where workflows break, where tools are misused, and which instructions need improvement, making them valuable evidence for refining an agent’s harness.