Multiple studies find that – without exception – the reliability of autonomous agent workflows decays geometrically over many steps. There are no examples of complex, reliable software created autonomously by agents – or anything even close.
On a per step basis, humans are – of course – equally unreliable. And yet, somehow, there are many examples of complex, reliable software created by humans.
Despite being just as fallible, humans are able to stabilise reliability in a way LLM-based agents can’t – despite all our best attempts to make them “self-correcting” or “self-healing”.
I think the missing piece of the jigsaw lies in my simplified model of the reliability of a step in development workflows:
R = 1 – (1 – C)(1 – P) Where C is the probability of it being correct, and P is the probability of any errors being caught before they propagate and compound – before the wheels start to wobble.
I think what we’re looking for is in 1 – P. If P represents the distribution of errors that get caught, then 1 – P represents the errors that fall outside that distribution. These are the errors that weren’t anticipated – the tests the agent didn’t write, the rules the linter didn’t check, the things the model wasn’t trained on.
The state of the art in self-correcting systems can only correct problems if they recognise them as problems. They can only self-correct if the problem is in P.
It’s the gap between P and 1 that causes the instability in autonomous workflows, and there is always a gap. P = 1 is infinitely difficult to achieve on any non-trivial problem.
Humans must have some capability that compensates – some ability to recognise and then adapt to the new and the unfamiliar, and with sparse data to go on.
Cognitive science calls it “learning”.