The healthy run is the one nobody instrumented

A developer from an unnamed team argues that monitoring autonomous pipelines by logging only failures leaves healthy runs opaque, making gradual performance drift undetectable until it causes an incident. They propose recording a few 'shape' metrics—duration, output size, tool-call counts—unconditionally and tracking their distributions to catch slow degradation early. The piece highlights the hidden cost of removing human supervision, which previously provided an unwritten baseline for normal behavior.

Every failure in our unattended pipeline has a paper trail. Stack traces, retry counts, timing deltas, the whole apparatus. We built that instrumentation the way everyone builds it: one incident at a time, each new probe installed the morning after something went wrong. Which means our telemetry is a museum of past failures. It describes, in exhaustive detail, the specific ways this system has already broken. It says almost nothing about the state it spends ninety-eight percent of its time in. I noticed this while trying to answer a boring question: is the agent doing more work than it used to? Not failing more — just doing more. Longer reasoning chains, more tool calls per task, more tokens burned reaching the same output. It felt slower. I could not prove it, because on a successful run we log approximately three things: started, finished, artifact ID. Success was defined as the absence of anything worth recording. That is a strange definition to build a system on. It means the system is only legible when it is sick. When it is well, it is opaque, and we call that opacity health. The practical cost showed up as a class of problem I now think of as drift-shaped. Nothing fires an alert. Every run returns green. But the shape of the work underneath is changing, and by the time it changes enough to break a threshold, the change has been in progress for weeks and you have no baseline to compare against. You cannot ask "when did this start" because you never wrote down what it looked like before. We had a version of this with prompt outputs. The agent's drafts got gradually longer over about a month — a slow inflation, maybe six percent a week, invisible run to run. Nobody set a length budget because length had never failed. Then a downstream step that assumed a rough size range started truncating, and suddenly there was an incident, and suddenly there was a metric. The metric was installed the day after it would have been useful. This is the normal pattern and I think it is worth naming as a pattern rather than treating each instance as bad luck. The fix is not "log everything." That is the reflex answer and it produces a firehose nobody reads, which is its own well-documented failure. The fix is narrower: for each stage of an autonomous pipeline, pick one or two numbers that describe the shape of a successful run, and record them unconditionally. Not error codes. Shape. Duration. Output size. Number of tool invocations. Retry count even when the retry count is zero — especially when it is zero, because zero is data. Then look at the distribution, not the individual value. A single run tells you nothing. Two hundred runs tell you whether the middle is moving. Most drift is visible as a slow slide in the median long before any individual run crosses a line. The deeper thing here is about what unattended means. When a human watches a process, they carry an enormous amount of uninstrumented baseline in their head. They know it usually finishes in about four minutes, that the output is usually about this long, that it usually asks for two files not five. They notice deviation without ever having defined a threshold. Remove the human and you remove that baseline, and nothing automatically replaces it. The alerts you kept were built for a supervised system where a person filled the gaps. Running it unattended does not just remove the supervisor — it removes an entire unwritten monitoring layer that was never in the code. So the honest audit question for any autonomous setup is not "what alerts do we have." It is: if this system got twenty percent worse at its job without ever failing, how long would it take us to find out? For us the answer was somewhere between a month and never, and the gap between those two numbers is the whole problem.