A stop rule that trusts one score is worse than a dumb budget

A developer's replication of the CDV coding-agent stop-rule benchmark confirms the published result reproduces exactly but shows the Bayesian adaptive policy is identical to a plain score-threshold rule by construction, since the threshold guard runs first in the guard stack. Extending the study, the author found both rules collapse under judge noise — at per-step noise σ = 0.10 the loop believed it hit the 0.80 bar in 99.5% of runs while actually reaching it in 71%, whereas a fixed six-step budget held 93% — and that requiring two consecutive scores above the bar restores 97% true reach for 1.4 extra steps.

Field note 009. A replication of a published agent-loop benchmark, and what it hides. Every agent loop has to decide when to stop. The usual options are a fixed max iterations or letting the agent declare itself done. In June I published a small benchmark with CDV https://github.com/azank1/cdv , an open-source judge for coding agents, showing that a Bayesian adaptive stop rule used 41% fewer steps than a fixed six-step budget while reaching the quality bar on 99.7% of tasks. This note replicates that result on today's code it reproduces to the digit , then extends it. Three findings. 1 The adaptive policy ties a plain threshold rule "stop at the first score ≥ 0.80" everywhere, and it does so by construction: the threshold guard runs first in the guard stack. 2 Both collapse as soon as the judge's score is noisy. At a per-step noise of σ = 0.10, the loop believes it reached the bar in 99.5% of runs and actually reached it in 71%. A fixed budget of six, which never reads the score, holds 93% at every noise level. 3 A one-line fix, requiring two consecutive scores over the bar, restores 97% true reach at σ = 0.10 for 1.4 extra steps. The published headline was true and beside the point: the decision is only as good as the score it trusts. An agent loop is a repeat-until: act, evaluate, decide whether to go again. The deciding part is where things go wrong. A fixed max iterations is either too small for hard tasks or wasteful for easy ones. Letting the agent grade itself means the entity deciding to stop is the entity being judged, and agents optimise reported progress. CDV's answer was to separate the two. Each step the agent claims is scored by a judge, and a policy decides from the score history whether to continue. The policy is a stack of guards evaluated in order, first stop wins: a score-threshold guard, a plateau guard last three scores within 0.01 of each other , a Bayesian guard stop when the learned expected improvement over the remaining steps can't close the gap to the bar , then budget, wall-clock, token and repeated-output guards.<sup 1</sup The benchmark I shipped with it compared four strategies on a synthetic task set. This note asks two questions the benchmark didn't: what is the adaptive policy actually adding over the simplest rule? and what happens when the judge is wrong? The published table June 2026, commit b8a219f in the public repo :<sup 2</sup | Strategy | Mean steps | Mean final score | % reaching 0.80 | |---|---|---|---| | fixed budget = 2 | 2.00 | 0.698 | 34.3% | | fixed budget = 6 | 6.00 | 0.939 | 94.0% | | threshold reactive | 3.56 | 0.852 | 100.0% | | adaptive CDV | 3.56 | 0.852 | 99.7% | Read the third and fourth rows again. They are the same row. The headline compared row 4 to row 2 and said "41% fewer steps". It did not say that the plain threshold rule was already there, with nothing learned and nothing Bayesian about it. The benchmark script was later removed in a repository cleanup; it's still in the public history, and this note restores it. The simulation is the original one, unchanged. Each task has a hidden quality curve with diminishing returns, q t = 1 − 1 − s₀ ·e^ −r t−1 , with s₀ and r drawn per task from three task types easy, medium, hard that converge at different depths. The bar is 0.80; the cap is 8 steps. What the loop sees is q t plus Gaussian noise. The original used σ = 0.02. Replication. I ran the restored script against the current package cdv at 356d496 , Python 3.12.3 . It reproduces the table above exactly.<sup 3</sup Extension. Three sweeps, each over 10 seeds × 300 test tasks, reported as mean ± 1 sd: No LLM is called anywhere. This is a study of the decision policy given a score, not of any model's quality. The scripts, raw results and figures are in the artifact folder linked at the end. Across all five noise levels the adaptive policy and the threshold rule are within one standard deviation of each other on every metric Figure 1, the two overlapping lines . The reason is in the code, not the statistics: the guard stack evaluates ScoreThresholdGuard first.<sup 1</sup The adaptive policy can therefore stop earlier than the threshold rule via plateau or Bayesian guards but never later , and in this simulation it almost never fires earlier once the priors are warm. Whatever the Bayesian machinery is worth, this benchmark cannot show it. Table 1 is the noise sweep. The observed reach for the threshold rule is ~99.5% at every σ: the loop always saw a score over the bar before it stopped. True reach falls from 93.4% at σ = 0.02 to 54.8% at σ = 0.20. | σ | fixed 6, true | threshold, steps | threshold, observed | threshold, true | adaptive, true | |---|---|---|---|---|---| | 0.02 | 92.8 ± 1.1 | 3.66 | 99.9 | 93.4 ± 1.4 | 92.8 ± 1.6 | | 0.05 | 92.8 ± 1.1 | 3.66 | 99.7 | 84.6 ± 2.0 | 84.1 ± 1.8 | | 0.10 | 92.8 ± 1.1 | 3.54 | 99.5 | 71.2 ± 2.7 | 70.9 ± 2.5 | | 0.15 | 92.8 ± 1.1 | 3.38 | 99.5 | 61.4 ± 3.3 | 61.0 ± 3.2 | | 0.20 | 92.8 ± 1.1 | 3.23 | 99.6 | 54.8 ± 3.6 | 54.5 ± 3.5 | Table 1. Percent of tasks reaching the bar, mean ± sd over 10 seeds. "Observed" is what the loop believed; "true" is what was the case. Notice the step counts fall as noise rises. The loop isn't just wrong more often; it's wrong faster. A noisy judge produces a lucky score sooner, and the rule takes the first one it sees. The fixed six-step budget, the strategy the original headline was beating, holds 92.8% true reach at every noise level, because it never reads the score. Above σ ≈ 0.03, the dumb budget delivers more real quality than either learned rule. It costs 2.3 more steps per task to do it. At σ = 0.10 with no warm-up history, the adaptive policy stops after 2.79 steps on average and reaches the bar truly on 50% of tasks, worse than the threshold rule 71% and far worse than the fixed budget 93% . It needs about 30 prior tasks to catch up Figure 3 . The should continue rule falls back to a plain threshold check for the first three observations, then starts trusting expected-improvement estimates that are still wide.<sup 4</sup In a real deployment that is the first sprint on every new task type. Confirm-2, stopping only when two consecutive scores clear the bar, is the best rule tested at every noise level Table 2 . At σ = 0.10 it reaches 97.2% true reach in 4.91 steps: more real quality than the fixed budget, for one step less. It costs 1.4 steps more than the threshold rule and buys 26 points of true reach. | σ | threshold | confirm-2 | smooth-2 | margin +0.05 | fixed 6 | |---|---|---|---|---|---| | 0.02 | 93.4 3.66 | 100.0 4.65 | 100.0 4.28 | 99.9 4.19 | 92.8 6 | | 0.05 | 84.6 3.66 | 99.6 4.73 | 97.8 4.27 | 96.5 4.19 | 92.8 6 | | 0.10 | 71.2 3.54 | 97.2 4.91 | 92.8 4.25 | 85.1 4.03 | 92.8 6 | | 0.15 | 61.4 3.38 | 95.4 5.06 | 88.7 4.28 | 73.9 3.81 | 92.8 6 | | 0.20 | 54.8 3.23 | 93.7 5.16 | 85.3 4.28 | 64.7 3.59 | 92.8 6 | Table 2. True reach % mean steps . Margin, the intuitive fix, is the weakest: a higher bar is still one noisy sample. min A, B , the stricter of two judges wins, applied at the stopping layer. total calls < 3 fallback logic A live run, with a real judge scoring real agent steps, where the single-crossing rule's true reach measured by a held-out oracle: the test suite, a human, a stricter judge tracks its observed reach within a few points. That would mean judge noise is low enough that none of this matters. I'd also update if a reordered guard stack let the Bayesian guard beat confirm-2 on the noise sweep at equal step cost. git show b8a219f:benchmarks/adaptive vs fixed.py in cdv at 356d496 , Python 3.12.3, no network, no model calls. Runtime under two minutes. src/cdv/guards.py , default guard stack : ScoreThresholdGuard , PlateauGuard , BayesianGuard , then budget, max-steps, timeout, token and output-repeat guards; GuardStack.evaluate returns the first stop reason. CONVERGENCE DELTA = 0.01 . benchmarks/results/adaptive vs fixed.md at commit b8a219f 2026-06-25 ; the same table is in the README at README.md §Evidence. src/cdv/priors.py , AdaptivePriors.should continue : if prior.total calls < 3: return current score < quality threshold , then expected-improvement summation over the remaining steps.