I want you to remember this formula. It’ll be on a blue plaque outside my house one day.
R = 1 – (1 – C)(1 – P) Where R is the reliability of a single step – a single model interaction – in an agentic workflow, C is the probability of the model’s prediction being correct, and P is the probability of errors being caught before they propagate and compound if it isn’t.
There are things we can do to improve C, but there’s only so much uncertainty we can squeeze out of the input, and physics predicts that LLMs are very unlikely to get much more reliable.
So it stands to reason that the key to extending agentic horizons lies in validating the output from each step before it becomes input to the next.
The E in C.R.E.S.S. stands for Empirical – contexts should contain validated information, observed in reality – and not raw model output. I liken LLM output to untreated sewage. It has to be carefully filtered to be made safe for drinking in future interactions.
It’s important to remember that even a high P – say, 0.99 – will lead to rapid collapse in reliability over multiple steps. Most development teams have never built quality gates good enough for reliable extended agentic autonomy.
It’s also important to remember that errors compound. An error at step N can become input context for step N+1, and C decays with each uncorrected error.
Studies confirm the highly improbable nature of reliable long-horizon agentic software development.
And while horizons have undoubtedly been getting longer – doubling roughly every 6-7 months since 2019, according to METR – at reliability levels we’d consider usable in software, they are still *very *short.
Once the wheels start to wobble…
The upshot of this is that – very likely – the human (Actual Intelligence) will have to stay very much in the loop, checking for unanticipated problems frequently, and learning throughout the process not just how to fix those problems, but how to detect them in automated quality gates in future.
Intelligence – real intelligence – seems to be the missing ingredient that proponents of software “factories” (such a category error of a misnomer) are overlooking. Yes, you *will *need to read the generated code. It’s either that or anticipate every problem in advance (good luck with that!), or wait until uncaught errors manifest far downstream when they’re much harder to diagnose and fix.
This is where my concept of feedback latency – the number of consequent decisions made before a root decision is validated – comes back to bite us. That gap where P < 1 – and it always will be – is where problems can leak into future steps in the process, and the longer we go without looking for unanticipated problems, the bigger the cost of addressing them.
In essence, the key here is maximising signal-to-noise in feedback about outputs – e.g. a test that fails after the agent made 1 change to the code instead of after 100 changes, and so on. Noise increases with feedback latency and signal degrades, with more and more uncertainty building up.
When we’re working with LLMs, uncertainty is not our friend. The probability of success decays with every additional step, and how fast it decays – and how soon you’ll need to step in – will be determined by C and P. And as models – and what we can squeeze out of them with context engineering – inevitably plateaus (it’s always an S-curve) it will rest increasingly on P.
Time to get good at quality gates!