Out-Of-Distribution Checks – How Human Intelligence Stabilises Agentic Workflows

A blog post on autonomous agent reliability argues that fully autonomous agentic workflows are "extremely improbable" to be reliable, citing arXiv paper 2510.27630 showing reliability tends to stabilise when humans are in the loop but not when agents work entirely autonomously. The author's STAAR model defines step reliability as R = 1 – (1 – C)(1 – P), where C is the probability of correct inference and P the probability of catching errors before they propagate, and notes that since P is always less than 1, out-of-distribution errors remain, causing R^N to decay over N steps. Using an example of an agent with R = 0.99 and a shippable threshold of Q = 0.95, the post asks how many steps the agent can take before reliability falls below threshold and how often a human must check for unanticipated errors, adding that LLMs do not learn and WHILE loops definitely do not learn.

Back to my “Special Theory of Autonomous Agent Reliability” STAAR . The reliability of a step a model interaction in an agentic workflow: R = 1 – 1 – C 1 – P Where C is the probability of correct inference, and P is the probability of catching errors before they propagate and compound. If P represents the distribution of anticipated errors – what we test for, what we lint for, what the model knows to look for etc – then, by definition, 1 – P represents out-of-distribution errors – we didn’t write a test for that, the linter doesn’t check for that, the model wasn’t trained on that etc. P is always < 1, and there will always be errors outside the distribution of our automated checks – be they deterministic or probabilistic. So, without checking for out-of-distribution errors, R<sup N</sup will inevitably decay over N steps, even at high values of R. What, or who, could spot unanticipated errors? What can operate outside of its distribution? Humans, that’s what We can know what we don’t know, recognise something as new or unexpected, and learn from very little data. This may explain why research into long-horizon workflows executed by agents show reliability tends to stabilise when humans are in the loop https://arxiv.org/abs/2510.27630 , but not when agents do the work entirely autonomously. The bottom line is that reliable fully-autonomous agentic workflows are extremely improbable, and all the credible data taps that sign. Let’s say we decide that an acceptable probability of reliability for shipping software is Q = 0.95 yes, that’s one-9’s reliability – aim low And let’s say our agent’s automated quality gates are pretty darn good, giving us R = 0.99. How many steps can the agent take before the probability of reliability, R<sup N</sup , falls below our shippable threshold? How often might we require someone to check for unanticipated, out-of-distribution errors to keep our software shippable? And if R<sup N</sup falls below the threshold Q, how long would it take to recover? Industry wisdom suggests that as feedback latency increases, the cost of correcting errors increases non-linearly. The longer uncaught errors propagate, exponentially the more they cost to fix, and the longer it takes us to reach Q again. The further we fall from shippable, the longer it takes us to climb back up. Naturally, there’ll be comments about “self-correcting” or “self-healing” agents. That would be to ascribe capabilities to LLMs that they simply don’t have. LLMs don’t learn. And WHILE loops definitely don’t learn.