{"slug": "a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget", "title": "A stop rule that trusts one score is worse than a dumb budget", "summary": "A developer's replication of the CDV coding-agent stop-rule benchmark confirms the published result reproduces exactly but shows the Bayesian adaptive policy is identical to a plain score-threshold rule by construction, since the threshold guard runs first in the guard stack. Extending the study, the author found both rules collapse under judge noise — at per-step noise σ = 0.10 the loop believed it hit the 0.80 bar in 99.5% of runs while actually reaching it in 71%, whereas a fixed six-step budget held 93% — and that requiring two consecutive scores above the bar restores 97% true reach for 1.4 extra steps.", "body_md": "*Field note 009. A replication of a published agent-loop benchmark, and what it hides.*\n\nEvery agent loop has to decide when to stop. The usual options are a fixed `max_iterations` or letting the agent declare itself done. In June I published a small benchmark with [CDV](https://github.com/azank1/cdv), an open-source judge for coding agents, showing that a Bayesian adaptive stop rule used 41% fewer steps than a fixed six-step budget while reaching the quality bar on 99.7% of tasks. This note replicates that result on today's code (it reproduces to the digit), then extends it. Three findings. **(1)** The adaptive policy ties a plain threshold rule (\"stop at the first score ≥ 0.80\") everywhere, and it does so by construction: the threshold guard runs first in the guard stack. **(2)** Both collapse as soon as the judge's score is noisy. At a per-step noise of σ = 0.10, the loop *believes* it reached the bar in 99.5% of runs and *actually* reached it in 71%. A fixed budget of six, which never reads the score, holds 93% at every noise level. **(3)** A one-line fix, requiring two consecutive scores over the bar, restores 97% true reach at σ = 0.10 for 1.4 extra steps. The published headline was true and beside the point: the decision is only as good as the score it trusts.\n\nAn agent loop is a repeat-until: act, evaluate, decide whether to go again. The deciding part is where things go wrong. A fixed `max_iterations` is either too small for hard tasks or wasteful for easy ones. Letting the agent grade itself means the entity deciding to stop is the entity being judged, and agents optimise *reported* progress.\n\nCDV's answer was to separate the two. Each step the agent claims is scored by a judge, and a policy decides from the score history whether to continue. The policy is a stack of guards evaluated in order, first stop wins: a score-threshold guard, a plateau guard (last three scores within 0.01 of each other), a Bayesian guard (stop when the learned expected improvement over the remaining steps can't close the gap to the bar), then budget, wall-clock, token and repeated-output guards.<sup>1</sup>\n\nThe benchmark I shipped with it compared four strategies on a synthetic task set. This note asks two questions the benchmark didn't: *what is the adaptive policy actually adding over the simplest rule?* and *what happens when the judge is wrong?*\n\nThe published table (June 2026, commit `b8a219f` in the public repo):<sup>2</sup>\n\n| Strategy | Mean steps | Mean final score | % reaching 0.80 | \n|---|---|---|---|\n| fixed (budget = 2) | 2.00 | 0.698 | 34.3% | \n| fixed (budget = 6) | 6.00 | 0.939 | 94.0% | \n| threshold (reactive) | 3.56 | 0.852 | 100.0% | \n| **adaptive (CDV)** | **3.56** | **0.852** | **99.7%** | \n\nRead the third and fourth rows again. They are the same row. The headline compared row 4 to row 2 and said \"41% fewer steps\". It did not say that the plain threshold rule was already there, with nothing learned and nothing Bayesian about it. The benchmark script was later removed in a repository cleanup; it's still in the public history, and this note restores it.\n\n**The simulation** is the original one, unchanged. Each task has a hidden quality curve with diminishing returns, `q(t) = 1 − (1 − s₀)·e^(−r(t−1))`, with `s₀` and `r` drawn per task from three task types (easy, medium, hard) that converge at different depths. The bar is 0.80; the cap is 8 steps. What the loop sees is `q(t)` plus Gaussian noise. The original used σ = 0.02.\n\n**Replication.** I ran the restored script against the current package (`cdv` at `356d496`, Python 3.12.3). It reproduces the table above exactly.<sup>3</sup>\n\n**Extension.** Three sweeps, each over 10 seeds × 300 test tasks, reported as mean ± 1 sd:\n\nNo LLM is called anywhere. This is a study of the decision policy given a score, not of any model's quality. The scripts, raw results and figures are in the artifact folder linked at the end.\n\nAcross all five noise levels the adaptive policy and the threshold rule are within one standard deviation of each other on every metric (Figure 1, the two overlapping lines). The reason is in the code, not the statistics: the guard stack evaluates `ScoreThresholdGuard` first.<sup>1</sup> The adaptive policy can therefore stop *earlier* than the threshold rule (via plateau or Bayesian guards) but never *later*, and in this simulation it almost never fires earlier once the priors are warm. Whatever the Bayesian machinery is worth, this benchmark cannot show it.\n\nTable 1 is the noise sweep. The observed reach for the threshold rule is ~99.5% at every σ: the loop always *saw* a score over the bar before it stopped. True reach falls from 93.4% at σ = 0.02 to 54.8% at σ = 0.20.\n\n| σ | fixed 6, true | threshold, steps | threshold, observed | threshold, **true** | adaptive, **true** | \n|---|---|---|---|---|---|\n| 0.02 | 92.8 ± 1.1 | 3.66 | 99.9 | **93.4 ± 1.4** | 92.8 ± 1.6 | \n| 0.05 | 92.8 ± 1.1 | 3.66 | 99.7 | **84.6 ± 2.0** | 84.1 ± 1.8 | \n| 0.10 | 92.8 ± 1.1 | 3.54 | 99.5 | **71.2 ± 2.7** | 70.9 ± 2.5 | \n| 0.15 | 92.8 ± 1.1 | 3.38 | 99.5 | **61.4 ± 3.3** | 61.0 ± 3.2 | \n| 0.20 | 92.8 ± 1.1 | 3.23 | 99.6 | **54.8 ± 3.6** | 54.5 ± 3.5 | \n\n*Table 1. Percent of tasks reaching the bar, mean ± sd over 10 seeds. \"Observed\" is what the loop believed; \"true\" is what was the case.*\n\nNotice the step counts *fall* as noise rises. The loop isn't just wrong more often; it's wrong faster. A noisy judge produces a lucky score sooner, and the rule takes the first one it sees.\n\nThe fixed six-step budget, the strategy the original headline was beating, holds 92.8% true reach at every noise level, because it never reads the score. Above σ ≈ 0.03, the dumb budget delivers more real quality than either learned rule. It costs 2.3 more steps per task to do it.\n\nAt σ = 0.10 with no warm-up history, the adaptive policy stops after 2.79 steps on average and reaches the bar truly on 50% of tasks, worse than the threshold rule (71%) and far worse than the fixed budget (93%). It needs about 30 prior tasks to catch up (Figure 3). The `should_continue` rule falls back to a plain threshold check for the first three observations, then starts trusting expected-improvement estimates that are still wide.<sup>4</sup> In a real deployment that is the first sprint on every new task type.\n\nConfirm-2, stopping only when two consecutive scores clear the bar, is the best rule tested at every noise level (Table 2). At σ = 0.10 it reaches 97.2% true reach in 4.91 steps: more real quality than the fixed budget, for one step less. It costs 1.4 steps more than the threshold rule and buys 26 points of true reach.\n\n| σ | threshold | confirm-2 | smooth-2 | margin +0.05 | fixed 6 | \n|---|---|---|---|---|---|\n| 0.02 | 93.4 (3.66) | **100.0 (4.65)** | 100.0 (4.28) | 99.9 (4.19) | 92.8 (6) | \n| 0.05 | 84.6 (3.66) | **99.6 (4.73)** | 97.8 (4.27) | 96.5 (4.19) | 92.8 (6) | \n| 0.10 | 71.2 (3.54) | **97.2 (4.91)** | 92.8 (4.25) | 85.1 (4.03) | 92.8 (6) | \n| 0.15 | 61.4 (3.38) | **95.4 (5.06)** | 88.7 (4.28) | 73.9 (3.81) | 92.8 (6) | \n| 0.20 | 54.8 (3.23) | **93.7 (5.16)** | 85.3 (4.28) | 64.7 (3.59) | 92.8 (6) | \n\n*Table 2. True reach % (mean steps). Margin, the intuitive fix, is the weakest: a higher bar is still one noisy sample.*\n\n`min(A, B)`, the stricter of two judges wins, applied at the stopping layer.`total_calls < 3` fallback logic\nA live run, with a real judge scoring real agent steps, where the single-crossing rule's true reach (measured by a held-out oracle: the test suite, a human, a stricter judge) tracks its observed reach within a few points. That would mean judge noise is low enough that none of this matters. I'd also update if a reordered guard stack let the Bayesian guard beat confirm-2 on the noise sweep at equal step cost.\n\n`git show b8a219f:benchmarks/adaptive_vs_fixed.py` in `cdv` at `356d496`, Python 3.12.3, no network, no model calls. Runtime under two minutes.`src/cdv/guards.py`, `default_guard_stack()`: `ScoreThresholdGuard`, `PlateauGuard`, `BayesianGuard`, then budget, max-steps, timeout, token and output-repeat guards; `GuardStack.evaluate` returns the first stop reason. `CONVERGENCE_DELTA = 0.01`.` benchmarks/results/adaptive_vs_fixed.md` at commit `b8a219f` (2026-06-25); the same table is in the README at `README.md` §Evidence.`src/cdv/priors.py`, `AdaptivePriors.should_continue`: `if prior.total_calls < 3: return current_score < quality_threshold`, then expected-improvement summation over the remaining steps.", "url": "https://wpnews.pro/news/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget", "canonical_source": "https://dev.to/azankhyder/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget-428c", "published_at": "2026-09-25 07:57:42+00:00", "updated_at": "2026-09-25 08:00:30.432360+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "developer-tools", "ai-research"], "entities": ["CDV", "GitHub"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget", "markdown": "https://wpnews.pro/news/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget.md", "text": "https://wpnews.pro/news/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget.txt", "jsonld": "https://wpnews.pro/news/a-stop-rule-that-trusts-one-score-is-worse-than-a-dumb-budget.jsonld"}}