Back to my “Special Theory of Autonomous Agent Reliability” (STAAR).
The reliability of a step (a model interaction) in an agentic workflow:
R = 1 – (1 – C)(1 – P) Where C is the probability of correct inference, and P is the probability of catching errors before they propagate and compound.
If P represents the distribution of anticipated errors – what we test for, what we lint for, what the model knows to look for etc – then, by definition, 1 – P represents out-of-distribution errors – we didn’t write a test for that, the linter doesn’t check for that, the model wasn’t trained on that etc. P is always < 1, and there will always be errors outside the distribution of our automated checks – be they deterministic or probabilistic.
So, without checking for out-of-distribution errors, R<sup>N</sup> will inevitably decay over N steps, even at high values of R.
What, or who, could spot unanticipated errors? What can operate outside of its distribution? Humans, that’s what! We can know what we don’t know, recognise something as new or unexpected, and learn from very little data.
This may explain why research into long-horizon workflows executed by agents show reliability tends to stabilise when humans are in the loop, but not when agents do the work entirely autonomously. The bottom line is that reliable fully-autonomous agentic workflows are extremely improbable, and all the credible data taps that sign.
Let’s say we decide that an acceptable probability of reliability for shipping software is Q = 0.95 (yes, that’s one-9’s reliability – aim low!)
And let’s say our agent’s automated quality gates are pretty darn good, giving us R = 0.99.
How many steps can the agent take before the probability of reliability, R<sup>N</sup>, falls below our shippable threshold? How often might we require someone to check for unanticipated, out-of-distribution errors to keep our software shippable?
And if R<sup>N</sup> falls below the threshold Q, how long would it take to recover? Industry wisdom suggests that as feedback latency increases, the cost of correcting errors increases non-linearly. The longer uncaught errors propagate, exponentially the more they cost to fix, and the longer it takes us to reach Q again.
The further we fall from shippable, the longer it takes us to climb back up.
Naturally, there’ll be comments about “self-correcting” or “self-healing” agents. That would be to ascribe capabilities to LLMs that they simply don’t have.
LLMs don’t learn. And WHILE loops definitely don’t learn.