By Mutagent AI Labs · July 15, 2026
We built a diagnostics agent. It reads a whole population of traces at once, and in ours it caught the agent sending messages that had skipped review, on paths so rare every check we kept still called it healthy.
Figure — The loop this piece is about: evaluation raises the alarm, diagnostics finds the cause, and the fix makes the agent better.
One of our own agents sends email on our behalf. Given an opportunity, it fetches the details, drafts a message, runs the draft through a review step, and only then sends. The review is not optional. It is the gate a send is supposed to wait behind. We were reading its traces, a few thousand of them, when one showed a send with no review anywhere in front of it. The message had gone out. Nothing had checked it.
The send succeeded. That was the problem.
The obvious reading. Every check we kept said the agent was healthy. The outcome check passed, because a send is a completed action and this one completed. No error surfaced, no user complained, no score dropped. On every dashboard we had, the run was indistinguishable from a good one. Grade an agent by whether its final action succeeds and a message sent without review is a success.
Evidence. The answer was in the path the agent took, not the outcome it reached. A healthy send here runs eight to eleven steps: fetch the opportunity, pull the details, draft the message, put the draft through review, send. This run was three steps long. It had fetched, and it had sent. There was no draft, and with no draft the review step had nothing to run on, so it never ran. The agent had written the message in one move and gone straight to the send, stepping around the two stages that exist to catch a bad one.
Finding. One cause with a row of consequences that each look survivable alone: no draft, so no review, so a three-step path, so a send that completes with every outcome check still passing and nothing downstream to react to. Each is true on its own, and each is invisible to a check that reads only the final action. The whole class of failure lives in the gap between the send worked and the send should have happened.
Notice where the cause was. Not on any dashboard, all of which were green, but in a step that was supposed to be in the trace and was not, on a path too rare for any sample of normal traffic to include. This is the failure a dashboard is built to miss. Finding it, across a whole population of runs, is the rest of this post.
The way most of us debugged software for the last thirty years is counterproductive on agents, and it fails in a way that feels like success. Software fails loudly enough that you rarely need a jury to know something broke. Agents fail politely, which is why they need evaluation at all.
Classic debugging is a search problem. Something broke, you have a stack trace, you follow it to a line. The trace is a rope and you pull it. That works because the failure and its cause live close together, and because the system is deterministic enough that reproducing the failure means re-running it. Agent failures have neither property.
The cause is far from the symptom. A vague sentence in the agent’s instructions causes a tool to be called with a bad argument, and the tool returns an empty result. The model reads that as “no data exists.” Forty seconds later it produces a confident, wrong, beautifully formatted answer. The stack trace, if one exists at all, points at the last thing that happened, and the last thing that happened is the least interesting thing that happened.
One trace cannot tell you whether it is a pattern. You open the trace of the failure someone complained about, find something wrong, fix it, and you have now fixed one trace. You do not know whether you fixed one case or a thousand. You do not know whether the thing you found is the cause or a bystander that shows up in healthy runs too. And you do not know whether the failure nobody complained about is eating four times as many requests.
Reading everything is not an option. A serious agent produces thousands of traces a day, thousands of steps each. Handing all of it to a model is a spectacularly expensive way to get a summary of the majority case, and the majority case is usually the case that works. So in practice you are blind to most of what your agent does, not by choice but by volume. A method earns its place only if it can pick the few traces worth reading, detect the signs of failure across the many it will never open, and fall back to statistics where reading cannot reach. That is the whole reason a method like this one has to exist.
So teams do the rational thing. They read the loudest trace, form a hypothesis that fits it, and ship a change. The symptom gets quieter and everyone moves on. Three weeks later the same root cause reopens as three tickets with three different names, and nobody connects them, because the first fix worked. That is debugging at scale, and it is not laziness. It is the only strategy available when you cannot see the population.
The two words get used interchangeably. They are not.
An agent gets evaluated on four things: the outcome, the trajectory, the tool outputs, and the context. Each layer sees something the others cannot, and each is blind in a specific way.
The outcome. Did the final answer come out right? Cheap, unambiguous, and the only layer your users experience directly. It is also a pass or fail on the last token. It cannot distinguish a right answer from a right answer for the wrong reason. It cannot see a failure that happened to cancel out, or the bug that will surface on the next input rather than this one.
The trajectory. Was the path the right path? Did the agent call the tools it should have called, in a sensible order, without loops and without skipping a step it needed? This layer sees detours and the tool that should have been called but was not. It has an expensive prerequisite: to say a path was wrong, you need to know what the right path was. Producing that reference is the hardest problem in the whole area, and we later put the cheapest reference to the test.
The tool outputs. Was each thing the agent got back actually correct? A tool can return a syntactically perfect response that is empty, stale, truncated, or an error wearing a success code. What this layer cannot tell you is whether the agent should have called that tool at all. A perfectly correct answer to the wrong question passes it every time.
The context. Did the agent have what it needed at each decision point: the instruction, the prior result, the document, the handoff from the step before? The hardest layer to judge, because it is counterfactual. You are asking whether information the agent did not have would have changed what it did, which is a question about a run that never happened.
One rule sits under all four, and we learned it the expensive way. A check is only as good as the reference it checks against, and the reference has to come from somewhere other than the thing being checked. We once watched two of our own quality checks agree on about 6,300 traces out of about 6,300. For a day that looked like validation. It was the opposite. One of the two had been derived from the other, so the agreement was arithmetic restating itself. It would have held just as perfectly if both had been wrong about every trace in the set. Perfect agreement between two measurements is not a result, it is a reason to go and read how they are computed. You cannot grade an exam with an answer sheet the student wrote.
Evaluation answers a closed question. Is this right? Yes or no, with a score, against a reference. It is the agent that raises the alarm: you run it continuously, gate deploys on it, watch it move. Diagnostics answers an open question. What happened, why, and where does it originate? It is the agent that finds the cause, and you would not want to run it on every request.
A score that drops six points is a fire alarm. It tells you the building is burning and nothing whatsoever about which room.
Here is the coupling between the two, the part that is easy to miss, and it is why they cannot be built as separate products that hand each other tickets.
Diagnostics does not replace evaluation. Diagnostics consumes evaluation, continuously, at every step of its descent.
Walking a trajectory backward is not a passive read. At each step you are running a small evaluation, and it is always one of the same three questions. Is this tool output correct? Was this step of the path the right step? Did the agent have the context it needed to make this choice? Each answer decides where you go next. If the tool output was correct and the step was still wrong, the cause is upstream in the reasoning. If the tool output was wrong, the cause is in the tool or in what was fed to it, and you descend there instead. If the context was missing, you stop asking about this step at all and go find who was supposed to provide it.
The descent is a sequence of judgments, and every judgment is an evaluation. Evaluation is the instrument. Diagnostics is the descent that carries the instrument down the trace. Remove the instrument and what is left is a system that reads traces and produces a fluent narrative, with no way to know which of its sentences the evidence actually supports.
The send we opened with reads back through the four layers with each doing a separate job. The outcome check passed: the send completed, and that is the whole of what it saw. A trajectory check catches the failure, because a path that sends without ever drafting or reviewing is wrong regardless of what the message said. The tool outputs check finds nothing to flag, and that is the point: every tool that did run returned a clean result. The failure is not in any output, it is in the steps that are absent. A context check names the location: the agent reached the send on a path that never produced the draft the review was there to read.
Four layers, four different questions. Not four scores. Only their intersection produces a name, and the name is what you can hand to someone.
Figure — The same four layers of a trace feed both agents. Evaluation returns a verdict that something is wrong; diagnostics takes the traces it fails and returns what broke, why, and where. The return arc is the coupling: every step of that descent runs another evaluation.
The design constraint pulls two ways: read enough to be right, cheaply enough to be repeatable. Every step below is a negotiation between them.
Figure — The pipeline. Everything above the deep read exists to make the deep read small: the cheap stages never conclude, they only decide where the expensive attention goes.
First, the free scan. Before any AI reads anything, a plain pattern scan sweeps every trace looking for error markers, latency outliers, cost spikes, and user feedback. No model calls, no cost. This pass never decides what is wrong. It decides where the expensive attention goes. That is triage rather than diagnosis, and keeping the line clean matters, because the moment a cheap scan is allowed to conclude, it will conclude the obvious thing.
An inventory comes next. Every distinct signal gets counted, and the benign ones get named as benign. This is what separates the method from a dashboard. A dashboard shows you what is frequent. An inventory asks what is frequent, what is harmful, and whether those are the same list. They are almost never the same list. Signals are then ranked by impact times prevalence, which produces the most counterintuitive behavior in the system: a failure that happens rarely but destroys the output ranks above one that happens constantly and changes nothing. Most monitoring gets this backwards, because counting is easy and consequence is hard.
Categorize what’s left. Traces are grouped with the traces they resemble: errors with errors, unhappy users with unhappy users, slow with slow. Hand an analyst a mixed group and you get a summary that describes the average of six unrelated failures and helps with none of them. Homogeneity is what makes a finding sharp.
After that, a bounded fan-out. The analysts are diagnostics agents in their own right, and several run in parallel, one to a group. Keeping that number small is a discipline, not a resource limit. If a problem appears to need fifty parallel investigations, the problem has not been framed yet, and fifty investigations will produce fifty opinions instead of one answer.
Each analyst works cheapest first, and this ladder is the whole cost story:
Figure — The escalation ladder. Most findings answer on one of the three free rungs and never reach the bottom. The rung where a model reads traces is the entire bill, which is why it is bounded and never skipped.
The deep read comes last, and it is bounded. When a model does read traces it starts with a sample and escalates the read depth in stages, each with its own time budget, up to a hard ceiling. Three rules keep that honest rather than merely convenient. A fresh investigation may never skip the deep read. Caps bound how much gets read, and they can never reduce it to zero. A system allowed to answer without reading will learn to answer without reading. What would count as an answer is fixed before the reading starts. A criterion set afterwards will be met by whatever you happened to find. The only defence is to write the criteria down first and open the evidence second. And coverage is reported. The report states how many traces were actually read and how much confidence that supports, in plain numbers. A finding that says “read four hundred traces, high confidence” and one that says “read twelve, hypothesis” should not look the same on the page. In most tools they do.
Every finding is stated in three parts: what broke, why it broke, and where it originates.
What broke is the symptom: wrong output, missing output, a loop, a latency spike, a cost blowout, a broken format, a fabrication. Why it broke is the mechanism: an underspecified instruction, a misused tool, a missing tool, overflowed context, a provider limit, stale data, something dropped in a handoff. Where it originates is the address: the instructions, a tool definition, the agent’s configuration, routing, the data upstream, the provider’s side, the harness, or the user’s input.
Three parts, because each one alone is a trap. “Wrong output” is a complaint. “The instructions are vague” is an opinion. “Wrong output, because the instruction does not say what to do when the field is missing, in the agent’s instructions” is a work item: you can hand it to someone, verify it, and recognize it when it comes back. Failures named this way are comparable, which means six months later you can ask whether this is the same failure as that one and get an answer rather than a memory.
Getting to the third part is the actual work, and there is only one method we trust for it.
Take the failure. Ask why. You will get an answer. Now ask why of that answer, and keep going, until the answer stops moving. The origin is the point where the next “why” either has no answer inside the system you control, or takes you into a different conversation entirely.
Here is the descent from the incident, written the way it actually ran:
The agent sent a message that should have been reviewed. Why? The review step never ran. Why? There was no draft for it to review. Why no draft? On this path the agent composed the message in one move instead of going through the draft step. Why did it take that path? The path is a rare, short one the agent only reaches on inputs the common paths never see. Why did nothing catch it? The send completed, so every check that reads only the outcome passed, and the path is too rare to appear in any sample of normal runs.
That last answer does not move. There is no further “why” that stays inside the system, only a different question about why a send was ever reachable on a path that never produced a draft. So that is the origin, and that is where the fix goes.
Figure — One cause, five consequences, and the false conclusion they add up to. Each consequence looks survivable on its own, which is the trap: the send stays reachable without a review until the one cause at the top is fixed. The dashed box is the false conclusion the five invite, the healthy verdict every dashboard returned.
Read that descent from the bottom up and it draws the figure. Fix any single consequence in isolation and the cause is still alive, because the send is still reachable without a review. Every dashboard returned the same healthy verdict, because every dashboard was reading the one layer that could not see the failure.
Three rules keep this from becoming storytelling.
Every link must cite evidence from an actual trace. A link you can only justify by reasoning is a hypothesis, it gets labeled as one, and it stays labeled that way. It is not promoted to a diagnosis because it sounds right and everyone is tired.
Stop when the answer stops moving, not when the answer sounds good. The common failure is stopping one step early, at the first answer that is actionable, and actionable is not the same as terminal. “The instructions are vague” is actionable: you could go rewrite the instructions this afternoon. Ask why the instructions are vague and you may find something that actually stops moving. Nobody specified the behavior for the case where the field is missing, because that case was never in the spec, because the spec was written against the happy path. Now you have an origin, and it is not in the instructions.
“Cannot be determined from these traces” is an answer. It is a better answer than a guess, and it is a real one only if it says why: which evidence would have settled the question, and why that evidence is not in the trace. Said that way, a dead end becomes a work item, usually a change to what gets recorded, and the next run of the same investigation can close what this one could not. A method with no way to return “unknown” will never return it. It will return something else instead, and you will have no way to tell which of its answers were the something else.
Evaluation re-enters here: each “why” is only answerable if you can judge the step below it, so the descent is only as good as those judgments.
One rule keeps all of this honest:
A diagnosis you cannot reproduce is a guess that agreed with you once.
So the reading is pinned. Same model version, no randomness, same findings on a re-run. That sounds like a minor technical detail. It is the difference between a method and a mood. If two runs disagree you do not have a finding, you have a generator of plausible sentences, and you cannot tell which of its outputs to trust, including the ones that happen to be correct.
Pinning the model is the easy half. The same discipline has to survive contact with people. In one of our own review rounds, four people working independently each reported that about fifty minutes had gone by. The clock said about twenty. Nobody was lying. A confident internal sense of how much work has happened is simply not a measurement, and four people agreeing did not make it one, for the same reason the two checks that agreed on every trace were not evidence. What kept the fiction out of the record was a mechanical rule: read the clock, write the number down. A model narrating its own certainty about a trace is doing exactly what those four reviewers were doing, and it will be exactly as convincing. This is why the confidence on a finding is not a mood the finding reports. It is the coverage number the deep read already reports, wearing a word.
Then a human. The system reads, names, and proposes ranked fixes. A person approves, and nothing changes without that. The system is good at seeing the population, which is what people are bad at. People are good at knowing that this particular service is being deprecated next month, which is what the system cannot see. Different jobs. The gate is where they meet.
The send we opened with was one trace, and one trace cannot tell you whether a thing is a pattern or an accident. So we turned the same reading on the whole population, every run that finished clean, and asked the harder question: not whether a run errored, but whether what the agent did was actually right.
The send-without-review was not a single accident. Across the population the same reading found it thirteen times, and those thirteen were a subset of seventeen confirmed violations it turned up in all. The seventeen split against the traffic two ways. By frequency, every one of them fell among the rarer, less-travelled runs rather than the most-travelled ones. By visibility, ten of the thirteen sends fell in an unclassifiable slice of traffic, roughly one run in ten. The failures lived where the traffic was thin, not in the bulk. To judge whether the agent was right, at scale, you need a reference for what right was. Where does that reference come from?
The cheap and appealing answer is frequency. Look at what the agent usually does, call the path it usually takes the correct path, and flag deviations. It requires no human judgment and it scales for free. Everyone tries it first. We tried it first.
The answer sheet rule says it cannot work before any measurement is taken: a frequency reference is the agent’s own behaviour, summarised and handed back as the standard the agent is graded against. It is the answer sheet the student wrote. The argument is free, so here is the measurement.
Two measurements killed it, both taken on a few thousand traces of our own agents.
Sampling by frequency is not a conservative choice. It is anti-correlated with what you are looking for.
Figure — Frequency vs correctness. A violation counts only where a second, independent look re-derived it.
We split those 6,319 traces by how common each trajectory shape was, then audited both groups for confirmed violations. The most frequent shapes, 2,114 traces, contained zero confirmed violations. The rare tail, 4,205 traces, contained all seventeen. Sampling by frequency, which is what “look at the typical case” means in practice, reads the 2,114 traces with nothing wrong in them and calls the agent clean. It is not. The tail holds every violation, and a frequency sample never reaches it.
This inverts the usual intuition about sampling. The common case is not representative of correctness; the tail is where correctness lives or dies. Common paths are common because they are the easy inputs, and the agent handles them.
Popular is not correct.
Figure — Popular is not correct. Each path was judged first by how often it occurred, then re-judged, blind, against what the agent should have done.
The second measurement is more direct. We took an audit set of 540 paths that the frequency reference had already classified. Then we re-judged them against what the agent should have done. That means someone deciding what the right action was at each point, blind to how often each path occurred. Roughly four in ten, 215 paths, changed classification. Both directions occurred. Patterns that were common but wrong had been sitting in the correct group the whole time, credentialed by nothing except repetition, and paths that were rare but right had been marked as deviations because they were unusual.
The most travelled path through your agent is just the path most travelled. Frequency is a description of habit, and habit is not a standard of correctness, in agents or anywhere else.
The sends that skipped a review step at least left a trace with something missing from it. The reclassified paths at least ran unusually. Both gave the reading something to catch. The worst behavior failure gives it nothing.
The agent returns a plausible wrong answer, fast and cheap, with no error and no complaint. The output is well formed. The path looks ordinary. No score drops, because an outcome check only ever sees the failures polite enough to announce themselves. That failure has no signature, and a pattern scan cannot match what leaves no pattern. It is found by reading, if it is found at all. That is exactly why the deep read can be bounded but never skipped, and why a run that produced zero findings is a suspicious result rather than a clean bill of health.
Figure — Outcome blindness. The outcome check sees only failures that announce themselves — an error, an outlier, a complaint. It misses a bad tool output halfway through, a drift in what the agent was doing, and a confident wrong answer with no error and no complaint. The proportion is illustrative; the shape is the point.
A named failure is a fixable one, and the name is the whole return on the reading. What broke, why it broke, and where it originates is not a report, it is a work item.
The send that skipped review. Named, it stops being a green run and becomes an address. The send was reachable on a path that never completed a review, so that is where the change goes: make the send wait behind a completed review, and record the review step so its absence is visible next time instead of silent. The check that had been blind, an outcome check that only ever saw a completed send, gives way to one that reads the path for the gate that should be there. The failure not one of our dashboards could see becomes the first thing the evaluation catches.
The reference the evaluation graded against. It got the same treatment. What the agent usually does gave way to what the agent should do, because we had measured the gap between the two and it was too wide to keep trusting the cheap one. A reference that called the rare tail deviant and the common bulk correct gave way to one that judges each path on its own merits. Neither change was large. Both were only possible once the failure had a name precise enough to fix.
Two failures, one discipline. Evaluation found the run was wrong, diagnosis named what and why and where, and the agent that ran next was the one that had stopped doing it. The same discipline, pointed at an earlier infrastructure incident, had put eight fixes in review inside a day and closed nine of ten issues in three days the week before.
The trade is real. Diagnosing first costs you a pass over the failures before a single line lands: you give up the patch you could ship this hour. What you buy is an upstream fix that does not reopen next month as three tickets under three different names.
That trade is only worth it when the cause is genuinely upstream of the symptom. For a typo it is not. Fix the typo. The trade earns out when the symptom is a liar, and in agent systems the symptom is usually a liar.
Diagnosis is not search, it is not a dashboard, and it is not a chat window pointed at your logs. Search finds the trace you asked about. A dashboard ranks by how often things happen, and what that ranking is worth is exactly what the frequency audit measured. A diagnosis names what happened, why, and where it started, and it earns each of those by judging the trace at every step of the way down.
The point of the name is never the name. It is the run after it. A cause with an address is one you can remove, and the agent that comes back is the one that stopped producing the failure. The middle step, the diagnosis, is the one teams skip, and skipping it is what turns the next fix into a guess.
Three changes are worth making before you adopt any method, including this one.
Instrument for diagnosis, not only for outcomes. You cannot walk a trajectory you did not record. If all you keep is the final answer and a pass or fail, the descent has no floor to stand on. The expensive way to learn this is to discover that the evidence for last month’s failure was never written down. Keep the steps, the tool calls with their arguments and their returns, and what the agent was holding when it decided. Keep them for the runs that succeeded too, because the population is the point.
Require a named failure mode before a fix lands. What broke, why it broke, where it originates, in one sentence, on the pull request, before the diff. This is the cheapest possible version of everything in this post: no tooling, no budget, one rule. A change that cannot name its failure mode is a patch on a symptom, and it leaves nothing behind that the next person can check.
Judge more than the outcome. The outcome check is the one you already have, and outcome blindness is the list of what it cannot see. Add the layers that catch what it misses: the trajectory that caught the send with no review, then the tool outputs and the context. They cost more than a pass or fail on the last token. What they buy is the failures that a pass or fail was never going to show you.
Every hard problem left is the same problem: how do you evaluate each of the four layers well enough to trust a diagnosis built on it? The outcome layer is cheap and close to solved. The other three are open. Judging a trajectory needs a reference for the right path, and the cheapest reference, what the agent usually does, is the one we just watched fail. Judging the tool outputs and the context well enough to catch a confident wrong answer, the failure with no signature, still comes down to reading, and reading is the thing we are trying not to do at population scale.
Which puts sampling at the center. The failures live in the tail, and a sample drawn by frequency is drawn from exactly the wrong place. A sample that represents your traffic does not represent your failures, and drawing one that does, without reading everything first, is the frontier: ways to evaluate the trajectory, the tool outputs, and the context as cheaply and as trustworthily as we already evaluate the outcome.
Diagnose first. Name the origin. Then fix exactly that, with your hand on the approval.
More on how we build agent diagnostics, evaluation, and improvement cycles at mutagent.io.
Diagnose and Evaluate Before You Mutate was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.