{"slug": "building-software-that-can-prove-agents-wrong", "title": "Building Software That Can Prove Agents Wrong", "summary": "A developer building a coding-agent workflow that handles triage, planning, implementation, verification, and pull requests found that implementation became the least worrying step, while verification — deciding what evidence proves a task complete — became the bottleneck. The developer argues that application design, not just the agent harness, determines what an agent can verify: a checkout flow where an agent only sees a \"Payment successful\" toast cannot detect double charges, duplicate orders, or duplicate events, whereas a system exposing deterministic resets, structured order state, balance diffs, retries, and traces allows far more autonomy. The essay concludes that the system being built supplies some of the sensors for the harness building it, turning development into a feedback loop.", "body_md": "EssayI\n\n# Building Software That Can Prove Agents Wrong\n\nWhat changes when implementation becomes cheaper than verification?\n\nRecently I've been building a workflow where coding agents take a task through triage, planning, implementation, verification, and finally opening a pull request. As the workflow and models got better, the thing that surprised me was that implementation stopped being the part I worried about most.\n\nThe agent could make code changes surprisingly fast. The harder part was getting it to verify those changes and decide what evidence was enough to consider the task complete.\n\nA verification step could exist in the workflow and the agent could run the application in a browser, reproduce the behavior, execute tests, and return screenshots or other evidence... but that raised a question I hadn't initially designed the workflow around:\n\nWhat kinds of mistakes was the agent actually capable of detecting?\n\nConsider a payment checkout change. An agent adds a new payment path, runs the application, completes checkout, presses \"Pay\", and then sees a toast: \"Payment successful\".\n\nThat is success evidence but it's a weak one. The UI can say \"success\" while we have two orders persisted. A retry logic can charge twice. An event can be emitted twice. The browser verification only proved one thing: the happy path looked right. It didn't prove the absence of several failures that actually mattered for us.\n\nOnce I started looking at agentic workflows this way, I stopped asking only:\n\nCan the agent use the product?\n\nI started asking:\n\nWhat kinds of mistakes can this product actually expose to the agent?\n\nThat question stopped feeling like a workflow-design problem and started looking like an application-design problem: the product itself was shaping what the agent could verify.\n\n## [§ 1](#some-systems-are-easier-to-prove-wrong)Some systems are easier to prove wrong\n\nTo make it easier to understand, imagine the same checkout feature in two different projects.\n\nIn the first, an agent can open the application and click through the flow but to understand what happened underneath, someone has to inspect a database manually, search through logs, reconstruct state, or explain how a retry should behave.\n\nIn the second, the agent can reset the application to a deterministic state, execute the payment, inspect structured order state, assert a balance diff, retry the operation, and query the trace.\n\nThe user-facing experience might be identical. The model might be identical. But the amount of autonomy you can safely give the agent isn't.\n\nOur first instinct is to attribute that difference entirely to the agent harness: the tools, context, constraints, evaluators, and feedback surrounding the model. Harness engineering has become a useful way to think about coding agents: the tools can include application UIs, logs, metrics, traces, and isolated runtimes, especially when human QA becomes the bottleneck.\n\nBut the harness doesn't have access to all the information it needs, and eventually, it reaches the application boundary where the application itself determines what can be observed and monitored. It determines whether an important state is observable, whether operations can be done through scripts, whether some specific scenarios are reproducible, and whether failures / errors contain enough data for the agent actually act on.\n\nThe system (application) being built supplies some of the sensors for the system (harness) building it.\n\nThat is the part I find interesting.\n\n## [§ 2](#development-starts-to-look-like-a-feedback-loop)Development starts to look like a feedback loop\n\nThat sensor layer matters because once the agent can act on what the application exposes, the work starts to look less like a handoff and more like a feedback loop.\n\nA simplified view of the old workflow might look like:\n\nWith a more robust coding agent / workflow, more of it can become a loop:\n\nIn my workflow, that meant the agent could change the code, run the product, verify the change, collect evidence, determine if it looks correct or not and try again.\n\nThis looks less like one-shot code generation and more like a control problem. And a control system needs useful sensors.\n\nFor a coding agent, the sensors are not just tests; they are the things that let it observe what happened: types, runtime state, logs, metrics, traces, browser output, tests, and independent evaluators. It also needs ways to act on the system (the actuators) such as: browsers, CLIs, APIs, and scripts.\n\nWhat I kept running into, though, was a more specific problem: verification was only as good as the information the software exposed. If the agent cannot see a state, event, or invariant, it cannot check it either.\n\nThis overlaps with what Birgitta Böckeler has called harnessability: how well a codebase supports this kind of agent-driven work. But I'm using that lens more narrowly here: not just whether the agent can work in the codebase, but whether the application exposes evidence that can cheaply and independently prove the agent wrong.\n\n## [§ 3](#the-verification-layer-has-to-be-able-to-see-the-bug)The verification layer has to be able to see the bug\n\nThat sounds obvious, but it's easy to miss when designing an agent workflow: adding more ways to verify work only helps if those tools can expose the failure types we actually care about.\n\nI mean, a screenshot can tell the agent that a toast rendered correctly, but it cannot tell when an operation was idempotent. A browser agent can complete checkout successfully without noticing that two orders were persisted. A green test suite can still pass if the agent wrote both the implementation and the tests from the same mistaken interpretation of the requirement.\n\nThis pattern shows up beyond checkout flows. Achint Mehta[arxiv.org/abs/2608.28795](https://arxiv.org/abs/2608.28795) recently measured a version of this effect across more than a thousand agent-generated web applications where different verification mechanisms helped with different failure modes: startup checks found apps that wouldn't launch, screenshots exposed visible UI errors, and performance problems only showed up when the evidence measured performance.\n\nThe main question I take from this isn't: \"*Did the agent verify its work?*\" It's: \"*Could the evidence available to the agent have revealed the relevant failure if it existed?*\"\n\nThat is a much harder standard.\n\n## [§ 4](#build-software-that-makes-incorrect-states-cheap-to-expose)Build software that makes incorrect states cheap to expose\n\nThat standard changes how I frame verification: there is a subtle difference between asking an agent to confirm that its code change works and asking what evidence could prove that it doesn't work. The first encourages confirmation while the second forces us to be explicit about the failure.\n\nUsing the checkout requirement as an example: *\"Clicking Pay should create exactly one order and deduct the correct amount from the balance.\"*\n\nA weak loop looks like:\n\nBut a stronger loop would look like:\n\nThe second loop isn't stronger because it contains more steps. It's stronger because those steps expose more ways for the implementation to be verified as wrong.\n\nThat changed the question I started asking myself:\n\nCan this software make important incorrect states cheap to expose?\n\nWhen I started applying that question to the agent loop, I kept coming back to a set of boring but useful properties:\n\n- Deterministic, reproducible scenarios mattered because the agent could get back to a known state instead of debugging whatever the last attempt left behind.\n- Explicit rules that must always hold mattered because they gave the agent a concrete condition to check against.\n- Queryable runtime state mattered because the UI often only showed the end result, not the state transition.\n- Structured failures mattered because the agent needed a clear failure reason before it could choose the proper fix.\n- Fast isolated environments mattered because every slow verification step becomes part of the time we're waiting for the agent feedback.\n\nAnd, to be clear, none of these are new engineering ideas; they overlap with testability, TDD, observability, and operability. What changed for me was their economic value: once an autonomous agent consumes the feedback loop, these properties lower the cost of each failed attempt and make it safer to let the agent keep iterating.\n\n## [§ 5](#evidence-speed-becomes-part-of-system-design)Evidence speed becomes part of system design\n\nWe already build for testability, observability, and operability, but the agent loop changes the economic value of those properties: they determine how quickly the agent can get useful evidence back.\n\nA trace is no longer only something an engineer opens after production gets broken. It can be an input to the agent. A CLI is no longer only for developers. It can also be used by the agent to set up state, perform an action, and inspect what happened during the execution.\n\nShopify's work[shopify.engineering/back-to-native](https://shopify.engineering/back-to-native) on mobile agents is a good example of this design pressure: once implementation is fast, the cost of getting evidence back starts to be the main part of the loop. The problem they describe wasn't that agents were too slow at writing code. It was that the feedback loop around the code was too slow: an agent could make a change in seconds, then spend minutes waiting on simulator interaction and to verify that the goal was achieved.\n\nThey also describe making business logic runnable headlessly and exposing it through a CLI so agents could inspect state, navigate, and perform operations without paying the simulator cost on every iteration.\n\nThe CLI is not the interesting part because it's something that have been around forever. The interesting part is that evidence speed became important enough to change the shape of the system.\n\n## [§ 6](#what-this-looks-like-in-practice)What this looks like in practice\n\nTo be honest, I don't think this requires inventing an \"agent architecture\". The practical move is simpler than you should think: audit the system through the eyes of an agent that needs evidence capable of showing that its own change is wrong.\n\nCan it cheaply create a known state?\n\nIf reproducing a bug requires a human to create accounts, click through onboarding, wait for asynchronous jobs, and explain what happened, the feedback loop is expensive. We should have deterministic fixtures, seeds, snapshots, and resettable environments. It brings that cost down.\n\nCan it inspect the state that actually matters?\n\nA UI / screenshot exposes only the final state of a much larger state transition. Having structured access to specific state, events, traces and diagnostics lets the agent check what changed, not just what was shown on the screen.\n\nAre important business assumptions easy to test?\n\n\"This should never create two orders\" helps, but a check that runs automatically and fails when two orders exist gives the agent evidence instead of something it needs to guess.\n\nCan behavior be tested without unnecessary UI work?\n\nThe browser is a valuable sensor and actuator, especially for requirements that are actually visible to the user, but some questions are better answered through more deterministic ways like a CLI, API, headless mode, or domain test helper. Using the product through a browser and debugging the system are different tasks that require different ways of working.\n\nAre failures machine-legible?\n\nAn error message saying \"Something went wrong\" is bad feedback for a human and even worse for an autonomous loop. If we want to reduce the amount of ambiguity and interpretation for our agents, we should ensure we have structured errors, traces, and explicit failure reasons.\n\nHow quickly can the loop restart?\n\nSlow builds, shared environments, manual authentication, flaky setup, and hard-to-reset state aren't just developer-experience problems anymore. That's basically what adds real friction to how quickly an agent can learn from a failed attempt, adjust its plan and perform the next move.\n\nAnd finally:\n\nWhat important claims still have no reliable mechanical check?\n\nThose gaps should be explicit and the point here isn't to automate every judgment. It's to know where the evidence ends.\n\nThe difference now is that these practices affect not only maintainability and developer experience, but how much implementation work an agent can safely do before a human has to step in.\n\n## [§ 7](#more-verification-isnt-necessarily-more-trust)More verification isn't necessarily more trust\n\nIt's important to notice that more verification doesn't necessarily mean more trust. This is just another trap.\n\nSuppose an agent:\n\n1. Interprets a requirement\n2. Writes the implementation\n3. Writes the tests\n4. Runs those tests\n5. Reviews the code\n\nIt looks like several layers of verification but they can all contain the same mistake. If the agent misunderstood the requirement at step one, it can faithfully propagate that misunderstanding into both the implementation and its tests. Five checks created based on one mistaken assumption may be weaker than one independent rule.\n\nI still think that coverage still matters but that's why a useful verification has at least two dimensions:\n\n- Coverage: could this check observe the failure?\n- Independence: is this check based on the same assumption that might be wrong?\n\nThe independence part is about where the check comes from: a pre-existing rule that must always hold is more independent than a test generated after implementation. Queryable runtime state is useful for the same reason: it can show what actually changed, rather than what the model believes its code did.\n\nAnd sometimes the only verification layer that can answer the question is a human.\n\nThis is why I don't think the goal should be to create \"self-verifying\" agents in some absolute sense but it's to give them stronger ways to prove themselves wrong before requiring human judgment.\n\n## [§ 8](#where-humans-still-matter)Where humans still matter\n\nThat leaves a narrower, but still important, role for humans: judging claims the system cannot turn into reliable evidence.\n\nSome software claims have clear checks an agent can verify:\n\n- Was exactly one order created?\n- Was the rule still true after the change?\n- Did this request exceed the latency threshold?\n\nOthers don't:\n\n- Is this interaction actually good?\n- Is this what the user meant?\n- Is this risk acceptable?\n- Does this feel trustworthy?\n\nAdding another agent doesn't magically make those questions deterministic.\n\nSo the point here isn't zero human involvement. It's using humans where human judgment provides information the agent cannot get and shouldn't infer.\n\nThat distinction matters more as agents get faster at implementation. Because if an agent can generate a 1,200-line change faster than a senior engineer can read it, asking the engineer to figure out whether the change is correct (based on the diff) is a poor scaling strategy.\n\nOver time, I expect more reviews to start with evidence:\n\n- What behavior did the agent test?\n- What rules were checked?\n- What state transitions were observed?\n- What failure cases did the agent try to reproduce?\n\nAnd, maybe most importantly:\n\nWhat was the agent unable to verify?\n\nThe code still matters, but reviewers should also look at the evidence, not only inspect the diff line by line.\n\n## [§ 9](#the-application-boundary-matters)The application boundary matters\n\nWith all that being said, if reviews are going to start from evidence rather than diffs, the next question is where that evidence comes from.\n\nIf we take a look at the first wave of improvements in coding agents, we'll realize that it came from better models. And a lot of the current improvements of the second wave are coming from better harnesses: context, tools, instructions, evaluators, browser control, isolated environments, and feedback loops.\n\nThese tools matter a lot, but eventually the harness reaches the application boundary and has to ask the application a few questions:\n\n- What happened?\n- What state changed?\n- Did this rule still hold?\n- Can I reproduce this scenario?\n- Can I perform this operation without manually navigating six screens?\n\nThis doesn't mean every product needs a dedicated API for agents. A more observable system can create coupling, security risk, and maintenance cost, so an agent-facing surface is only worth building when the behavior it helps inspect is important enough to justify that cost. Sometimes, the right answer is an existing test, trace, CLI, or human review.\n\nBut the design pressure is still real: software is increasingly being modified by systems that operate through feedback. The better systems can expose its real state to agents, the more capable the agent becomes without changing the model at all.\n\nThe model didn't improve. The environment did.\n\nThat change points us to a different question:\n\nHow easy is it for the software to prove that the agent is wrong?\n\nThe best software for coding agents may not be the software that is easiest to generate. It may be the software that is easiest to prove wrong.", "url": "https://wpnews.pro/news/building-software-that-can-prove-agents-wrong", "canonical_source": "https://www.rafael.md/writing/building-software-that-can-prove-agents-wrong", "published_at": "2026-09-21 20:59:49+00:00", "updated_at": "2026-09-21 21:24:06.604382+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-products"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/building-software-that-can-prove-agents-wrong", "markdown": "https://wpnews.pro/news/building-software-that-can-prove-agents-wrong.md", "text": "https://wpnews.pro/news/building-software-that-can-prove-agents-wrong.txt", "jsonld": "https://wpnews.pro/news/building-software-that-can-prove-agents-wrong.jsonld"}}