# What 80% AI-written test pipelines actually cost > Source: > Published: 2026-07-15 09:00:00+00:00 The first time I heard someone say their AI now wrote 80% of their tests, I asked the obvious question. Eighty percent of what? After 20 years building and leading test automation for consumer-scale platforms, my honest answer turned out to be eighty percent of the *typing*, not eighty percent of the *engineering*. The remaining twenty was where the work still lived. Budgeting for two percent of leftover effort was the mistake. When the real number was closer to thirty, that gap was the difference between a pipeline that shipped and one that quietly built up a queue of half-trusted features nobody could rely on. This piece is about that gap. As an independent research project on LLM-augmented testing methodology, I built a six-stage agentic pipeline that takes a design in Figma and produces running tests in WebDriverIO, connected end to end over the [Model Context Protocol](https://modelcontextprotocol.io). It works. It has been useful. And the parts that broke surprised me, because they were not the parts the hype cycle tells you to worry about. The pipeline runs six stages in sequence, each owned by a different agent, with every handoff crossing MCP. Six-stage agentic test pipeline: design capture → requirements writer → ticket opener → code generator → test-case writer → automation generator. Each stage carries an MCP handoff and a provenance stamp. The end-to-end trace links a pull request back to a Jira ticket, a requirements section and a Figma frame. Each artifact is stamped with the agent that produced it, the model it used and the inputs it was given. MCP is the boring middle that makes any of this work. The cliché is that MCP is “USB-C for AI”: one open protocol, any tool. Like most analogies, it is about eighty percent right. The part that matters is the eighty: I do not have to write a custom adapter for every system the agent talks to. One MCP server per tool and every agent talks to all of them the same way. **Typed handoffs between agents are my own architecture, layered on top of MCP rather than provided by it.** Each agent writes a typed artifact the next agent reads. Each handoff is logged with provenance. When something went wrong six stages in, I could replay the chain. Without that discipline, a multi-agent pipeline is a debugger’s worst day. You know the test plan is wrong. You cannot tell whether the mistake came from the Figma read, the requirements interpretation or the ticket scaffolding. With it, I could point at exactly which stage went sideways and which inputs it was looking at when it did. The pattern lives in a [public MIT-licensed reference implementation](https://github.com/SuneetMalhotra/agent-harness) for any reader who wants to run it. **The sixteen-minute number is the marketing number.** I ran the full chain end to end in about sixteen minutes on a synthetic net-new screen, Figma in, automation suite out. That repeated across my runs; it is not a demo trick. But sixteen minutes is the part of the story most fun to tell and least useful to learn from. It is what gets quoted in the all-hands. The hours that come after, when a human reviews each handoff, are where the work actually lives. The failures that stalled my pipeline were rarely the ones I expected. I expected hallucinated APIs. I got them: the agent confidently called endpoint names that sounded right but did not exist. I expected sparse-spec-in, sparse-spec-out, where a Figma frame with no annotations produced a requirements doc with vague acceptance criteria, every time. I expected locator drift, the common UI-automation failure mode where a renamed component silently breaks an entire test suite. There is solid [outside writing on non-determinism in tests](https://martinfowler.com/articles/nonDeterminism.html) covering this whole family of failure modes, and the agent inherited every one. What I did not expect, and what kept the pipeline down longer than any of the above, was the plumbing. The model backend timed out under load. It lost credentials silently and started returning empty strings, which the agent then read as confidence. A duplicate consumer on a shared long-poll API endpoint produced an HTTP 409 conflict that broke delivery without throwing anything visible. One unguarded exception inside one agent aborted a whole shared scheduler run and took the other agents in the registry down with it. The single worst incident cost me three hours to find. An environment variable had silently rotated overnight; every agent in the fleet was returning structurally valid but semantically empty requirements docs; the downstream stages were dutifully generating tests against nothing. None of those are model bugs. They are infrastructure. The agent literature, which is what I went looking through when I started this work, mostly does not talk about them. The fix was not better prompts. It was [circuit-breaker-style](https://martinfowler.com/bliki/CircuitBreaker.html) review checkpoints between stages and what I now call **the four-guard discipline**: four small guards I consider non-negotiable on any unattended agentic pipeline. The bulkhead pattern from microservices is the most consequential. An unhandled exception inside one agent can no longer abort the shared run; the offending agent fails fast with a structured error and the others keep going. Paired with that, a pure-data fallback ensures a model timeout produces a deterministic output explicitly marked as degraded mode, rather than an empty string the next stage will misread as confidence. A single-owner lease sits on every shared external endpoint, the cure for the duplicate-consumer incident that ate one of my Sunday afternoons. The cheapest guard was the last to arrive: a one-line synthetic canary every agent has to produce a known correct response to before any real work begins, so a credentials rotation or silent backend failure trips an alert before downstream stages have generated artifacts against garbage. None of these guards is novel. They are textbook stability patterns at a new boundary: the seam between the LLM agent and the rest of the system, which most of the existing agent literature still treats as a solved problem. Here is the part the demo videos leave out. Even when the pipeline works, the human time per stage does not go to zero. Human review time per ticket across five pipeline stages: code review 60-180 min, automation review and flaky-fix loop 30-90 min, ticket architecture and sequencing 30-60 min, test data and environment 15-30 min, requirements review 20-30 min. Net: the human still spends 20-30% of the original effort, almost all of it reviewing rather than creating. **Net of all that, the human still spends twenty to thirty percent of the original effort, almost all of it reviewing rather than creating.** The pipeline saves seventy to eighty percent, not ninety-eight. The trap is budgeting for the two percent you do not save. When does this kind of pipeline make sense? In my experience, when the Figma is richly annotated and acceptance criteria are clear up front; when there is review capacity to absorb the work the pipeline shifts onto humans; when the stack is well represented in the training data; and when the feature is net-new rather than a deep edit of legacy code. When does it not? When the design lives on a whiteboard. When the integration touches old code with hidden contracts. When the path is regulated or safety-critical. When there is no senior reviewer who can hold the line. When the work is exploratory and writing the spec is the actual point of the exercise. Teams I have seen succeed with agentic pipelines budget for the rework explicitly, staff the review queue and treat the saved hours as capacity for harder problems rather than headcount they can release. Teams I have seen struggle did the opposite: declared victory at the demo and quietly accumulated a backlog of half-trusted features the next quarter had to clean up. The right unit of measurement is not how much the pipeline generates. It is how much of what it generates a human still has to touch before you would ship it. Call it **the 80/20 rework rule**: measure the rework, not the generation. The teams that get the rework number right are the ones whose AI investments compound. The teams that stop counting at the headline percentage are the ones that own the cleanup six months later. **This article is published as part of the Foundry Expert Contributor Network.**__Want to join?__