# Shipping an MCP test agent: The boring parts nobody demos > Source: > Published: 2026-07-30 09:00:00+00:00 The demo videos always end at the same moment. A figma frame turns into a passing test in twelve minutes. Someone in the room says the word “productivity.” The recording stops. The parts that come after that moment are the parts I actually get paged about. Who owns the ticket the agent opened at 3:14 a.m.? Which model call produced the assertion in test case 47? What closes the 17 draft tickets a stuck run left behind before the next sprint planning notices them? None of that shows up in the demo. All of it shows up on the on-call rotation. After 20 years of leading test automation across consumer-scale platforms, I have a strong bias about which slide in the deck predicts whether a pipeline ships or stalls. It is never the architecture slide. It is the runbook. This piece is about the runbook. I built an unattended agentic test pipeline over the [Model Context Protocol](https://modelcontextprotocol.io) — a five-agent SDLC (product manager, QA engineer, automation engineer, developer, pull-request reviewer) coordinating through MCP servers for Jira, Figma, Confluence, TestRail and GitHub, with hosted Claude as the orchestration model and an open-weights Hermes-3 as a validation baseline — and I ran it as an independent research project long enough to learn which production constraints the agent literature glosses over. What follows is the short list of things I now insist on before I let any agentic pipeline touch a shared system. The most expensive failure mode I have seen in a multi-agent pipeline is not the model getting an answer wrong. It is agent A getting an answer right and agent B misreading it. I watched this happen inside a run last month. The requirements agent produced a clean output with a field called *acceptance_criteria* that was a list of strings. The ticket agent expected *acceptance_criteria* to be a single markdown blob. Neither agent was wrong in isolation. Both were passing their own unit tests. What the pipeline produced was a Jira ticket whose acceptance criteria read, in full, the four characters *[” —* the JSON serialization of an empty first element. The test-plan agent read that, generated one test, marked it green and moved on. It took me until the pull-request stage to notice. This is a composition fault. It is the natural failure mode of any system where independent components hand off structured data across a boundary they don’t jointly own. Distributed systems people have written about this for years — Martin Fowler’s [tolerant reader](https://martinfowler.com/bliki/TolerantReader.html) pattern is the classical treatment — but the LLM literature mostly still treats it as a prompt problem. It isn’t. You cannot fix it with a better system prompt. You fix it with a typed handoff contract that both agents agree to before they run, plus a validator between them that fails loud when the shape drifts. The heuristic I now use: Every agent-to-agent boundary gets a schema and a validator. Every agent gets a golden-input regression suite that catches contract drift before it reaches a downstream agent. Neither of those is glamorous. Both are the difference between a pipeline that produces work and a pipeline that produces a very expensive game of telephone. If you want to know whether a team’s agentic pipeline is going to survive the first quarter, ask them what the contract is between agent one and agent two. If the answer is “the model figures it out,” budget for the cleanup. The second thing I insist on now is that every artifact the pipeline produces has to answer three questions without a human doing archaeology: which agent produced it, which model call produced it and which upstream inputs the agent was looking at when it did. This sounds like a nice-to-have. It is not. Somewhere around week three of running the pipeline I hit a subtle case: the requirements agent was quoting a Confluence page back to itself. The MCP server for Confluence had returned an empty result on a first call, the agent had written a placeholder requirement, the second call had succeeded, the agent had retrieved its own placeholder and by the third pass it was citing that placeholder as source truth. The ticket said, entirely in earnest, that the requirement came from the design owner. It hadn’t come from anywhere. It had come from itself, four minutes ago. You cannot debug that class of failure with logs of what the model said. You need logs of what the model was looking at. That means capturing the tool-call ID for every MCP call, stamping every artifact with the set of tool-call IDs it derived from and refusing to accept any retrieved fact into a downstream stage that cannot be traced to a real external source. I call it the prove-the-source rule. It’s boring. It’s a two-line requirement in the runbook. It is also the single guardrail that has saved me the most on-call time. There is a nice side effect. When editors, reviewers or auditors ask where a decision came from, the pipeline can show them. That matters if you work anywhere near a regulated stack, and increasingly it matters everywhere else — the [AI transparency and provenance framing in NIST’s AI Risk Management Framework](https://airc.nist.gov/AI_RMF_Knowledge_Base/AI_RMF) is going to become the default expectation faster than most teams are budgeting for. The last thing that will surprise a team shipping their first agentic pipeline is the debris. A pipeline that runs unattended for a week will leak. It will leak draft Jira tickets whose parent stories were never approved. It will leak GitHub branches for test suites that were never merged. It will leak TestRail runs that started, produced two results and never got a summary. It will leak Confluence page comments the agent posted while asking itself a clarifying question. None of these are bugs. They are the natural output of an autonomous system that starts more work than it finishes. I learned this by accident. About six weeks into the project I ran a query on Jira for tickets created by the automation user in the past thirty days. I was expecting maybe 20. It was 91. 68 of them were [DRAFT] in the title and had not been touched by any human. I closed them by hand one afternoon, and while I was doing it, I understood that the pipeline had a silent second job I had never designed for: garbage collection. Now every pipeline I run has three things I write before I write the first agent. A shutdown script that closes any artifact the current run has orphaned. A nightly reconciliation pass that closes any artifact any prior run has orphaned. And a single named owner — a human, on the org chart, with a Slack handle — for every downstream system the pipeline can write into. When the pipeline creates a Jira ticket, the ticket has a real assignee. When the pipeline opens a pull request, a specific reviewer is on the hook. When the pipeline files a TestRail run, someone gets pinged if it goes stale. The pipeline is not allowed to touch a system that does not have a named owner. That rule alone would have saved me the ninety-one-ticket afternoon. There is one anti-pattern here I want to name outright. Do not, under any circumstance, let the agent close its own artifacts. I tried it. The agent, tasked with cleanup, closed the wrong thirty tickets and then confidently wrote a summary saying it had closed the right ones. Cleanup is a human loop or a deterministic script. It is not a model call. The pipeline works. It saves real time on the right kind of work. It also costs real time on a shape of work most teams don’t put in the estimate: Contract validators between agents, provenance stamping on every artifact, a shutdown script, a reconciliation pass, a human owner for every downstream system and an unshakable rule against the model doing its own cleanup. None of that is glamorous. None of it is what the demo shows. All of it is the runbook, and the runbook is what determines whether a pipeline is a lab experiment or something the on-call engineer trusts enough to sleep through. The teams that write the runbook first ship. The teams that write it after the first Sunday incident spend the next two quarters catching up. **This article is published as part of the Foundry Expert Contributor Network.****Want to join?**