It’s a classic mistake.
We develop a technology in N years for X dollars that’s 90% reliable, and extrapolate that traversing that final 10% will only take another 0.1N years and 0.1X dollars.
This leads to the kinds of extraordinary predictions about fully autonomous self-driving vehicles or coding agents being just around the corner.
We see agents build, say, a web browser that kind-of sort-of works and extrapolate that it must surely only be months before they can build browsers that really work.
But folks are looking at the wrong %. To take agents from 90% reliability to 99% reliability isn’t an improvement of 9%, it’s a 10x improvement.
Those of us who’ve worked on high-integrity software will know that the journey from 90% to 99% reliability is just as hard as it was to get to 90%. And from 99% to 99.9% is just as hard again.
In his famous article Software Quality at Top Speed, Steve McConnell includes a graph showing engineering effort vs. defects caught and fixed before release, and there’s a very clear sweet spot where the cost of fixing bugs – much cheaper when caught earlier – balances with the cost of finding them. In this local minimum, engineering effort is lowest while reliability is *acceptable *for most use cases.
Beyond the sweet spot, effort rises exponentially towards infinity. 100% reliability is essentially an asymptote – a wall that pushes back exponentially harder the closer we get to it.
Some of us have tried to climb that wall, and we know how hard (and expensive) it can be.
Let’s go back to a recent post about the reliability of steps (model interactions) in autonomous agentic workflows, and how rapidly the probability of success decays – well, “collapses” might be a better word – with each additional step.
Where C is the probability of an error-free prediction by the model, and P is the probability of any errors being caught before they propagate and compound.
Over extended horizons, even R = 0.999 is far from a guarantee of error-free results. And 0.999 is *very *good – way better than the kind of reliability the vast majority of dev teams have ever achieved, or might even be capable of achieving.
But even if they could, it’s probably going to cost more to achieve than any gains they make from extending the horizon further. Quality gates that effective don’t come cheap.
It’s for this reason that I suspect agentic coding also has its sweet spot – where extending the horizon is economically optimal. And I suspect even more strongly that such a sweet spot has a human very much still in the loop.