cd /news/ai-agents/the-harness-isnt-the-moat-what-you-l… · home › topics › ai-agents › article
[ARTICLE · art-171920] src=pub.towardsai.net ↗ pub= topic=ai-agents verified=true sentiment=· neutral

The Harness Isn’t the Moat. What You Learn From It Might Be.

A harness change that shortened older tool results and responded to stalled work raised the mean per-task fail-to-pass fraction from 28% to 49% and complete solutions from 43 to 72 on a 169-task SWE-bench Verified cohort, according to an August 2026 paper by Sydney Lewis, "Same Model, Different Harness: Different Coding-Agent Results." Lewis held the model and task fixed across three coding benchmarks, using a 20,480-token context window and a fixed 480-second limit, and the same treatment improved both measures across three additional models without model-specific retuning. The result indicates the improvement came from how the system managed context pressure rather than from one model's behavior, an argument the author extends to ask whether such harness engineering can become a durable competitive advantage for companies building on models they do not own.

recommendation

When agent context fills, compress older tool results and intervene on repeated or stalled steps; the source measured fail-to-pass rising from 28% to 49% across four models without retuning.

more in playbook: Designing agent systems
by read8 min views6 publishedNov 18, 2026

From Opus 5.5 to GPT 3.2 or Gemini 3.8, we’ve seen the AI industry spend the last few years competing on models. And every new release brings another improvement or round of conversations about reasoning, context windows, benchmark scores, inference speed and model size. BUT… we know that model capability alone does not determine whether an AI product works. A model can perform well on benchmarks and still struggle to complete a relatively simple task when placed in a real environment. It might lack access to the right information, fail to use a tool correctly, lose track of what it has already done or mistake an unsuccessful action for a completed one.

As an engineer, I experience these problems while building AI agents myself. In some cases, the problem was how the system around the model supplied context, exposed tools, managed state and handled the result of an action. Changing the model would not necessarily have fixed any of that.

To solve this, Harness Engineering was introduced. It is the system that connects a model to the environment in which it operates, and determines what the model can access, what it can do and how the application knows whether it has done the job correctly.

OpenAI’s Codex team has written about the growing importance of this engineering layer, particularly as developers move from writing code directly to building environments in which AI agents can work.

But I am interested in a question that goes beyond whether a harness makes an agent perform better: Can the harness become a source of competitive advantage for the companies building AI products, and if so, how long can that advantage last?

The first question is relatively straightforward:

Can two companies using the same model build products that perform very differently?

The answer is yes, and the difference often comes down to how each company has built the system around that model.

A company might have access to the same underlying model as its competitor, but that does not mean both products can complete the same tasks with equal reliability.

One might have better access to relevant information, more carefully designed tool interfaces, stronger state management and a way to verify whether an action succeeded. The other might rely on the model to work through those problems as they arise.

The difference becomes visible when the systems encounter real tasks rather than carefully prepared demonstrations. A model might be capable of completing a task, but the surrounding system determines whether it has the information, permissions and feedback needed to do so consistently.

The August 2026 paper by Sydney Lewis, Same Model, Different Harness: Different Coding-Agent Results, supports this. It examined what happens when the model and task remain fixed but the harness changes.

Lewis compared two configurations of the same harness across three coding benchmarks. The control configuration supplied the full conversation in chronological order. The treatment configuration shortened older tool results as the context filled and responded to repeated or stalled work.

On a 169-task SWE-bench Verified cohort, with a 20,480-token context window and a fixed 480-second limit, the treatment increased the mean per-task fail-to-pass fraction from 28% to 49%, while complete solutions increased from 43 to 72.

The same treatment also improved both measures across three additional models without model-specific retuning. That suggests the improvement came from how the system managed context pressure rather than from exploiting the behaviour of one particular model.

For companies building on top of models they do not own, this is significant. They may not be able to compete with the companies training the underlying models, but they can still improve what those models accomplish. The system around the model is something they can design, measure and improve. But better performance on its own does not establish a moat. If every competitor has access to the same techniques, tools, and architecture, the advantage may not last long.

Most of the components used to build a harness are not particularly difficult to reproduce. Competitors can copy tool integrations, rebuild retrieval pipelines, and increasingly rely on orchestration patterns available as open-source libraries. A competitor can study an existing product, work out how its components fit together and build something similar. AI tools are also making it easier to inspect and reverse-engineer software.

But reproducing how a system is built is different from understanding how it behaves when people rely on it.

Consider an agent handling insurance claims. Its architecture might be straightforward enough to replicate: connect a model to policy documents, expose tools for retrieving customer records and submitting claims, and add a workflow for cases that need human approval. What a competitor cannot immediately reproduce is what the original team has learned from running that agent.

After handling enough claims, the team may discover that certain documents are frequently missing, that particular policy clauses cause interpretation problems, or that an external service regularly returns incomplete responses. They begin to understand which cases the agent can handle, where it needs additional checks and when a human should take over.

That experience only becomes useful when it changes how the system works. A recurring problem with missing documents might lead to an additional validation step. Repeated errors from an external service could prompt a change to the tool interface or a fallback procedure. Cases that require human judgement can become part of an evaluation dataset used to test future changes.

Over time, the harness starts to reflect what the team has learned from operating the product. A competitor might reproduce the same architecture and connect the same model, but it would still have to discover many of these problems and work out how to address them.

I’ve encountered a smaller version of this while building my own bots. The fixes that lasted were usually tied to something I had seen the system get wrong. I could identify the failure, document it and turn that observation into a rule or a change in the workflow. Other changes, based on what I thought should work, were less dependable. The difference was whether I was responding to an actual problem or making an assumption about how the system should behave.

This is where operational knowledge can become a source of advantage. As a team addresses recurring failures, it can make the agent more reliable in the environment it was built for. And as customers begin to depend on that reliability, the product can become harder to replace. It may be connected to internal databases, permission structures, approval processes and reporting systems. A replacement would have to account for those integrations, reproduce the relevant workflows and demonstrate that it can handle the same tasks reliably.

There is a limit to this argument, though. Collecting logs and customer interactions does not automatically give a company an advantage. The team has to understand what those records reveal and use them to improve the product. Without that work, the data is just a record of what happened.

Well, not entirely because models will keep changing, providers will ship new versions, adjust pricing, improve reasoning and retire older models. A company that builds its entire product around the specific behaviour of one model can end up dependent on that provider’s roadmap.

So a well-designed harness creates some separation between the model and the business logic. It allows a company to evaluate different models, route tasks according to their requirements, use smaller models for simpler operations and reserve more capable ones for work that needs them.

Something to note also is that model providers are also building more of the infrastructure that surrounds their models. Features that once required custom engineering are increasingly becoming part of model platforms and agent development frameworks. As these components become easier to access, the cost of reproducing a basic harness falls.

So even if a company builds a better harness today, there is no guarantee that the same architecture will remain a meaningful advantage tomorrow. Because competitors can adopt similar tools, providers can absorb parts of the stack, and new models can change which engineering decisions matter.

This raises some harder questions:

My honest answer is that I am not sure the harness itself becomes a moat in the way the term is usually understood.

The defensible position may instead be what the harness gives a company access to: proprietary operational knowledge, distribution, customer relationships or integrations that are difficult to reproduce. In that case, the harness is not the moat. It is the machinery that helps a company build one somewhere else.

I would rather leave that question open than claim the evidence has settled it.

I don’t think the harness is automatically a moat. Its architecture can be studied and reproduced, and the tools used to build it will become more accessible. But operating an agent gives a team the opportunity to learn things that are not obvious from looking at its architecture. Whether that knowledge becomes a lasting advantage depends on how well the team turns it into a product that customers can rely on.

That leaves a more practical question: what does it actually take to build and operate a harness that works? That’s where the engineering deserves closer attention.

The Harness Isn’t the Moat. What You Learn From It Might Be. was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

── more in #ai-agents 4 stories · sorted by recency
── more on @sydney lewis 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/the-harness-isnt-the…] indexed:0 read:8min 2026-11-18 · —