A probability grid of chain-of-thought, read through Boyd's OODA loop lens

A new analysis by Adam Sohn, based on thousands of multiplication problems run through reasoning models including Claude Opus and Qwen3, finds that chain-of-thought reasoning follows the four phases of John Boyd's OODA loop—observe, orient, decide, act—and that reinforcement learning selects for error-catching behavior because a single slip makes the final answer wrong. The analysis, published on adamsohn.com, argues that the speed of iteration, not quality, determines success in both aerial combat and AI reasoning, and that technical debt compounds across cycles.

A probability grid of chain-of-thought, read through Boyd's OODA loop lens Boyd's loop During the Korean War, when both pilots saw each other, they merged head-on at a closing speed near a thousand miles an hour. The pass was over in seconds. What followed was a turning fight, each trying to end up behind the other. John Boyd, an Air Force pilot who spent his career on those seconds, described what a pilot does as four steps run over and over. Observe, orient, decide, act, then look again, because the other aircraft moved during the decision. He argued that the pilot who gets through that loop faster wins, even when his individual decisions are the worse ones. Jeff Atwood in his blog, Coding Horror https://blog.codinghorror.com/boyds-law-of-iteration/ , took it from Roger Sessions https://web.archive.org/web/20070211180233/http://msdn2.microsoft.com/en-us/library/aa479371.aspx and compressed it into a slogan. Speed of iteration beats quality of iteration. Software took that slogan as sprint advice. Iterate faster, ship smaller, fail early. That reading treats every cycle as a fresh start, and it misses what Boyd was pointing at. His fight has no fresh starts. The speed a pilot burns in one turn is still gone when the next turn begins, so a small disadvantage per cycle compounds into an unrecoverable one. Software calls it technical debt. Every shortcut one sprint takes slows every sprint after it, and slows them by more each time. A reasoning model works the same way. It reads every token in front of it, works out a probability for each token that could come next, picks one, and adds it to what it reads on the next pass. That is the whole mechanism, and nothing in it knows what a step is. What comes out is natural language, and that language is Boyd's list. Here is one run, in order. Okay, so I need to calculate the exact product of 2053896 and 30957123778 is observe. Alternatively, maybe use the calculator approach? But since I can't use a calculator, I need to do it manually is orient. In the run that came out wrong, floating an idea like that and then dropping it is the commonest move the model makes, and not one of them produces any work. Alternatively, use the distributive property is decide, the same word it just used to float an idea and drop it. Then comes the arithmetic, which is act and the only one of the four with no line to quote. Thirty passages of thinking later, Wait, let me check the addition steps again . That looks like a fifth step and it is not one. Checking the sum is observe again, and the second time it points at what the model wrote rather than at the problem it was handed. Boyd's pilot has to look again because the other aircraft moved while he decided. The model has to look again because it wrote something while it decided, and what it wrote is now part of what it reads. I ran thousands of multiplication problems through these models, and had Claude Opus label the thinking text https://adamsohn.com/reliably-incorrect/ of individual runs segment by expensive segment. The same patterns appear in models from different companies, and in coding tasks https://adamsohn.com/lambda-variance/ . Every one of them reads the problem back, chooses an approach, does the work, checks it, catches a mistake and goes back. No objective named those four phases, but that is not the same as nobody selecting for them. Qwen3 was trained by reinforcement learning https://arxiv.org/html/2505.09388v1 against thousands of query and verifier pairs, and that reward arrives only when the final answer is exactly right. A slip that survives to the end makes the answer wrong, and on a chain this long some slip is close to certain, so no amount of care reaches that reward on its own. Catching them does. That is what the training actually selects for. Both it and Phi-4-reasoning, the second model on this page, were handed <think tokens, which put a wall around the reasoning without saying anything about what goes inside it. And replications of that training keep finding base models already writing "wait" and "let me verify" before any of it ran, so it raised the weight on something that was there rather than inventing it. So the loop is a property of how these models were scored, not something in the architecture. Wherever a task has one exact answer and a long way to reach it, checking is the shortest route to that reward, and the loop follows. Where a wrong step is survivable, nothing selects for the habit and there should be no loop to find. Long multiplication sits at the first end of that range, which is why the traces here are so legible, and it is also the caveat on everything that follows. Normally a model would call a tool. Doing the multiplication by reasoning instead is expensive, and it burns the same GPU time as a complex coding task. To the model the two are the same work. What multiplication gives is control over the difficulty, the way a control rod gives control over a reactor. Move it and the reaction changes by a known amount. Every model here reaches for the same longhand method taught in school, one partial product for each digit of the second number, each shifted by its place, then all of them added up. So every extra digit adds an exact number of small steps rather than a vague amount more work. A seven digit number against an eleven digit one takes seventy seven of them. Against a twelve digit one it takes eighty four. None of this can be memorised. Every problem size gets a cell of its own, digits in one number against digits in the other, and one of the smallest cells is a single digit against nine. That cell alone holds eight billion different problems, and the model working them here has four billion parameters. The whole fourteen by fourteen grid holds 1028, which is ten thousand trillion trillion. There is nowhere to put the answers, so the model has to work them out. Both rows run the same loop. The difference is what the model does when it checks. In the first row it works the product two ways. The two ways disagree, and that disagreement is what saves it. It goes back, finds the slip, and finishes with the right answer. In the second row it checks by redoing the same sum the same way. It gets the same wrong number, finds no conflict, and carries that number to the end. A check only helps when it could have failed differently. The run that failed is the one to keep in mind. Every phase of the loop ran. The model checked its own work. The answer was still wrong. The rest of this page is about how often that happens, and at what size it starts. How I evaluated the models Frontier models are prohibitively expensive to run thousands of times, so I rented decent GPUs by the second on Modal instead. Before spending anything on the grid I ran smoke tests to find where the models begin to break and where they absolutely fail, so I had a known range to explore. There is no point paying to measure the sizes everything solves or the sizes everything misses. I wanted the band in between. Then I used cost-weighted Neyman allocation to work out how many runs each size actually needs. Low complexity always solves fast and settles in a few runs. High complexity always fails and settles just as fast. Medium complexity is the problem, because a model can grind away there for a long time and still land either way, so that is where most of the runs have to go. I put several models through all of this to find the two that came out closest, Qwen3-4B from Alibaba and Phi-4-reasoning from Microsoft. How much variance is there? Winning the battle does not require perfection. It requires correction that could have failed differently from the step it is checking. Just like the F-86 dogfights, a high rate of error is tolerable as long as something in the loop can catch it. Multiplication serves as a controlled dial to scale this complexity. Model failure is not a black box. The rate moves smoothly with size, even though any single run in the middle band is a toss-up, and the gradient marks the edge of what these models can do unaided. The surface below tracks that boundary. One axis counts the digits of a, the other the digits of b, and cell height reflects the proportion of runs that landed the exact product, with 4 × 4 representing a problem like 3,437 × 9,122. Sample sizes vary across cells. The uncertain middle earns more attempts. The surface redraws on each trial, which shows which rates have settled and which are still moving. Only an exact answer counts as a success. A wrong digit in the answer is a failure, and so is a run that runs away. Given room, a model will generate tens of thousands of tokens. Some were still working when the run was stopped, so whether they would have landed it is unknown, and paying to find out is prohibitively expensive. Non-converging runs are scored identically to incorrect ones, as both represent unusable output. The raw data retains all four distinct outcomes. The grid simply collapses them. Looking at these charts, it is easy to conclude these models are hopelessly limited. The top end does go to zero on the runs I did, at sample sizes too small to call it impossible. But that falloff hides an important structural advantage. Measuring where the model's reliability falls off, even as a probability rather than a verdict, gives us a way to forecast results. Where each one stops I expected the two models, Qwen and Phi, to fail on different problems, and the reason was a pattern I kept hitting. Opus 4.6 would write a query that caused a full table scan on 450GB of timeseries data, and the Claude models I used to review missed it every time. Regardless of how much prompting I did or how many specific instructions I put in code comments, it would go on making the same mistake. Handing the same review to Codex caught it far more often. That is a blind spot one vendor had and the other did not, on a problem, writing queries for timeseries analysis in Postgres, and it is the hypothesis this grid was built to test. I will keep chasing understanding it in the future. The method followed from the hypothesis. Probe until I find the two models that come out closest, then hand them one problem type whose complexity I can raise in small steps, and check which specific problems each one misses. Multiplication did not show it, and at 1.5 attempts per problem it was never going to. Both curves fall away over the same range and sit about a digit apart in each factor, and there is nowhere on the grid where Phi is ahead of Qwen by more than noise. My guess is that the same consistency holds for lambda calculus and other coding work, and that where models really do differ it comes down to what they were trained on rather than how capable they are. Some frontier models solve a Rubik's Cube and others that keep up with them on arithmetic cannot. I did not measure that here, and my read is that one company put it in the training set, as data or as a rubric. One thing weakens the comparison and it belongs here rather than in a footnote. Phi-4-reasoning learned to reason by supervised fine-tuning on 1.4 million reasoning traces generated by OpenAI's o3-mini. So these two models come from different companies but not from independent lineages, and nothing in this data can separate how much of their agreement is convergence from how much is inheritance. The loop has to be able to find out it was wrong With multiplication we had the benefit of a cheap verifier. That is what makes the grid possible. Every cell shows where the model stops being reliable, because we always knew the right answer. Solving complex coding problems isn't much different from solving complex multiplication problems, granted we have a cheap verification process. We shouldn't expect the model to solve every complex problem in one pass. Rather than stopping at an understanding of the limitations, we can decompose problems into manageable parts. The iteration process matters too. Observe, orient, decide, act. That is the purpose of the agent harness, which is the interface between the environment and the model. The input is the observation and the output is the action. The models are not coin flips. They are instruments with a working range, and inside that range the outcome is predictable.