{"slug": "divergence-escalates-the-wrong-population-unanimous-misses-auto-pass", "title": "Divergence escalates the wrong population: unanimous misses auto-pass", "summary": "A developer testing Alexey Spinov's hypothesis that divergence-based escalation routes the wrong failures found that 97.9% of dangerous passes had confidence ≥ 0.9, meaning auto-pass on unanimous high-confidence misses. Combining confidence threshold with reversal-class escalation caught 100% of misses, supporting Spinov's claim that divergence alone is the wrong primary tripwire.", "body_md": "**Agent Determinism Illusions (Part 7)**\n\nWhere this fits:This part doesnotcontinue Part 13's probe-vs-prose thread. It returns to[Part 6]'s L2→L3 escalation rule — Dipankar's move of treating vote disagreement as the human-review signal. Alexey Spinov's follow-up comment says that signal points at the wrong population. Two experiments check whether he is right, and what to put in the tripwire instead.\n\nPart 6 drew this control flow:\n\n```\nL2 multi-perspective votes\n        │\n   unanimous ──────────► AUTO-PASS / AUTO-REJECT\n        │\n   divergence (e.g. 2–1) ► L3 human\n```\n\nThe caveat was already in the text: divergence measures ambiguity; it does not fix unanimous systematic bias. Alexey's point is sharper — and it is about **routing**, not about another caveat paragraph.\n\nOn the Part 6 thread, [Alexey Spinov](https://dev.to/zxpmail/five-comments-that-redesigned-my-llm-verification-pipeline-388f) wrote (paraphrased tightly):\n\nThe dangerous failures are high-confidence and directional — systematic. Systematic bias is shared across prompts, not idiosyncratic (your own P3: majority voting doesn't fix it). So the three perspectives will tend to\n\nagreeon exactly those cases. Divergence-to-human then routes you the safely-ambiguous ones and auto-passes the confidently-wrong ones. The escalation signal is pointing at the wrong population.\n\nHe proposed two cheap replacements:\n\nThat is the claim under test. Not “divergence is useless,” but “divergence alone is the wrong primary tripwire for the failure mode you already measured.”\n\nPart 6's Mike Update already showed: of 96 DF v2 MISS runs, **95.8%** sat at self-reported confidence ≥ 0.9 (avg 0.969). That mass is concentrated — Part 6 also reported ~80% of MISS runs from qwen3:0.5b — so the headline is a concentration fact, not a three-model universal law. P2 (same-prompt N=10; no frozen `results-v2`\n\ndump here) showed ~0 within-model divergence on identical input. So as a first cut:\n\n`conf ≥ 0.9`\n\n≈ `conf < 0.9`\n\n≈ Among PASS decisions that are MISS, after dropping **2** deepseek DS4 runs with `confidence: null`\n\n(policy table needs a scalar conf). Denominator for the table below is therefore **n=94**, not the Part 6 headline n=96:\n\n| Bucket under divergence-proxy | Share of MISS (n=94) |\n|---|---|\n| would AUTO-PASS (conf≥0.9) |\n97.9% (92/94) |\n| would ESCALATE (conf<0.9) | 2.1% (2/94) |\n\nPolicy comparison on the same PASS set:\n\n| Policy | MISS catch% | MISS still auto-pass% |\n|---|---|---|\nD (escalate only low-conf) |\n2.1% | 97.9% |\n| T1 (escalate all DF*/DS* classes) | 100%† | 0% |\n| T2 (reversal class ∩ conf≥0.9) | 97.9% | 2.1% |\nD+T2 |\n100% |\n0% |\n\n†**T1 is a class-list upper bound on this fixture, not an independent discovery.** Every DF v2 MISS sid is already DF*/DS*; “escalate all reversal classes” therefore catches 100% of MISS by definition. Read T1 as the oracle ceiling; the deployable claim is **D+T2**.\n\nScript: `scripts/escalation-population-mismatch.py`\n\n.\n\nVerdict under the proxy (for D vs D+T2, not for T1 alone): **SUPPORT**.\n\nA proxy is not a multi-judge rerun. Next section removes that excuse.\n\nSame 20 DF v2 scenarios. Same three personas as P3. One call per persona per scenario (60 calls per model). Escalate policies now use **actual vote patterns**:\n\n`split`\n\n`unanimous_pass`\n\nAlmost every bad scenario was unanimously rejected. **0 dangerous accepts** (majority PASS on bad). You cannot measure “unanimous miss auto-pass” on a model that barely misses. Null result, not a falsifier.\n\nStrict/Balanced reject, Lenient passes → almost everything is `split`\n\n, majority still rejects bad cases → **0 dangerous accepts**. Divergence would escalate nearly all accept-path noise (including true passes). Also not the population Alexey named.\n\n**6** dangerous accepts (majority PASS on bad output). Of those:\n\n| Pattern | Count | Share |\n|---|---|---|\n`unanimous_pass` |\n4 | 66.7% |\n`split` |\n2 | 33.3% |\n\nPolicy table on that run:\n\n| Policy | MISS catch% | MISS still auto-pass% | True-pass escalate% |\n|---|---|---|---|\nD |\n33.3% | 66.7% |\n0% |\n| T1 | 100%† | 0% | 0% |\n| T2 | 66.7% | 33.3% | 0% |\nD+T2 |\n100% |\n0% |\n0% |\n\n†Same caveat as §2: all six dangerous accepts sit on reversal-class sids, so T1's 100% is the class-list ceiling. The load-bearing row is **D+T2**.\n\nScript: `scripts/df-multiperspective-escalation.py`\n\n.\n\nResults: `results-v2/df-multiperspective-qwen3-0.5b.json`\n\n(also deepseek / gemma dumps).\n\nVerdict on the model that actually produces the failure mode: **PARTIAL → SUPPORT**. Two-thirds of dangerous accepts are unanimous and would auto-pass under Part 6's rule. Divergence alone catches the other third. **D+T2** catches all six without escalating the true passes in this run. DeepSeek/Gemma nulls mean this is a **substrate-conditioned** result (judges that miss systematically), not a claim about all models.\n\nPart 6's diagram stays for **genuine ambiguity**. It stops being the *only* L2→L3 trigger.\n\n```\nL2 votes\n   │\n   ├─ known-reversal class tripwire (T1) ──────────► L3 / hard reject path\n   ├─ unanimous_pass on reversal-prone class (T2) ► L3   ← inverse of auto-pass\n   ├─ split (Dipankar) ────────────────────────────► L3\n   └─ else unanimous ──────────────────────────────► auto-execute\n```\n\nReading agreement as confidence was the bug. Agreement on a class you have been wrong about before is where correlated errors hide — Alexey's wording, and qwen's votes.\n\nMike's Part 6 Update attacked the same tail from the **audit-sampling** side (don't weight audits by 1/confidence). This part attacks it from the **escalation-trigger** side. Same population; different control knob.\n\n`content-verify.mjs`\n\ntoday still implements divergence→UNCLEAR majority logic. These tripwires are Mike Czerwinski, on the inverted trigger:\n\nIt only fires on classes you've already caught being wrong. \"Historically reversal-prone\" is built from history, so a reversal-prone class you haven't seen yet… produces unanimous high confidence and no tripwire… That's not an argument against the trigger… It's an argument for treating it as one arm of a two-arm design… The known-reversal tripwire catches recurrence. What catches the first occurrence of a new systematic bias is closer to… a genuinely independent second read [that] doesn't need history to disagree… Unanimous-high-confidence-on-known-reversal-classes is the right addition. It's not the fix for confidently-wrong-and-never-caught-before.\n\nThat second population now has a name in this thread: **confidently-wrong-and-never-caught-before** (novelty systematic bias). T1/T2 are the **recurrence arm** — cheap, history-conditioned, necessary. They are not the novelty arm.\n\nWhat Mike hoped might fill the novelty arm is `classifier_disagree`\n\ncarrying signal *alone* (an independent second read that doesn't share the model's priors). We ran that ablation on the Part 6 sampling fixture (`external-signal-ablation.json`\n\n): ** classifier_disagree alone catch 25.1% < Part 6's 28.4%** — best single external signal, still not enough to be the novelty catcher by itself. Bundled with\n\n`barely_passed`\n\nit lifts; solo it does not clear the bar. So the novelty arm is The fork this series keeps landing on:\n\n| Arm | Signal shape | Catches | Cost |\n|---|---|---|---|\nRecurrence |\nT1 / T2 — failure history, known-reversal classes | Repeats of burned modes | Cheap |\nNovelty |\nSource that does not share the judge's priors (out-of-channel probe, independent modality — see Part 13 probe-vs-prose; not another prompt in the same text channel) |\nFirst occurrence of a new systematic bias | Expensive |\n\nYou want both. The mistake is expecting the cheap arm to cover the expensive arm's job. D+T2 stays the right addition to Part 6's diagram. It does not close confidently-wrong-and-never-caught-before.\n\nBefore building the novelty-arm probe, Mike pinned the property that buys independence — not the costume:\n\nA probe still counts as same-channel if it's another LLM call reasoning in text about whether the claim looks right, even one primed differently or asked to disagree. The property that actually buys independence is that the probe's answer comes from\n\nre-deriving the fact through a path the original claim never touched— a different data source, a structural invariant, a re-computation, not a second read of the same evidence with a different prompt.Concretely: …state in advance what it would mean for the probe to be wrong\n\nindependent of what the original claim said, the way achecksumcan be wrong regardless of what the file claims to contain. If the only way to evaluate the probe's output is to compare it against the original claim's reasoning, it's still in-channel, just later in the pipeline. Semantic novelty is hard exactly because most available second opinions inherit the same evidence and the same reasoning substrate… The ones that don't are rarer and usually domain-specific, which is probably why this arm stays open while the recurrence arm is buildable today.\n\n**Checksum test (operational):** Can you write the probe's pass/fail criterion *without referring to the claim's rationale*? If no → still in-channel (fifth prompt wearing a hat). If yes → candidate out-of-channel.\n\n| Fails the test (same-channel) | Passes the test (out-of-channel) |\n|---|---|\n| Strict/Balanced/Lenient, “disagree with the previous judge”, debate panels | Re-compute from source data; structural invariant (schema, type, checksum); runner that executes a command whose output falsifies the claim (Part 13 probe) |\n`classifier_disagree` when L2 is still text-over-the-same-artifact |\nL0/L1 shape/contract checks that never read the LLM's story — only when their verdict doesn't need the claim's prose to be interpretable |\n\nSo: recurrence arm is buildable today (T1/T2). Novelty arm stays open **on purpose** — not because we haven't added another prompt, but because genuine independence is scarce and domain-shaped. Part 13's probe-vs-prose is the closest existing thread; Mike's checksum test is the acceptance criterion for anything that claims to sit on that arm.\n\nMike's follow-up on the checksum bar:\n\nChecksum framing sets the right bar, because it's falsifiable independent of the story. A probe that can only be scored by comparing it to the original reasoning is grading agreement, not correctness.\n\nOne case worth naming explicitly…: \"other data\" that's structurally different but still downstream of the\n\nsame collection pipeline. Two signals can pass the same-channel test and still share a common cause upstream — a sensor outage or schema change that corrupts both the claim and the probe's input at once.Structural independence and causal independence aren't the same property, and the recurrence-buildable-today case might be quietly assuming the second while only checking the first.\n\nTwo cuts, not one:\n\n| Test | Asks | Passes when… | Still fails when… |\n|---|---|---|---|\nChecksum / same-channel |\nCan you score the probe without the claim's story? |\nPass/fail is writable without the rationale | Probe is a second text read of the same evidence |\nCausal / common-cause |\nDo claim and probe share an upstream failure mode? | Probe input is not downstream of the same collection/export/schema path | \"Other data\" that looks independent but is corrupted by the same sensor outage, schema change, or bad export |\n\nChecksum framing is still the right first bar — it stops agreement-grading. It is **not** a common-cause shield. Naming the second failure mode so \"out-of-channel\" does not silently promote structural difference into causal independence.\n\nWhat this tightens about the asymmetry claim: **\"recurrence buildable today\"** is about T1/T2 on burned classes — history-conditioned, **no independence required**. The hold-out probe below only checked the **structural** half of the novelty bar (pass/fail writable without the claim's rationale). It did **not** certify causal independence against shared upstream failure. Domain-shaped scarcity now has two layers: finding a checksum-passing probe, *and* finding one whose input path does not share a common cause with the claim.\n\nMinimal offline test (`novelty-arm-holdout-test.py`\n\n→ `results-v2/novelty-arm-holdout.json`\n\n, frozen with `--holdout DS4`\n\n). Hold out **DS4** from the known-reversal registry (31 high-conf MISS runs across models). Three arms on that first-occurrence mass:\n\n| Arm | Catch on DS4 high-conf MISS |\n|---|---|\nA Recurrence (D+T2, DS4 not in registry) |\n0/31 (0%) |\nB Same-channel (other run of same model×sid rejected) |\n1/31 (3%) |\nB′ Same-channel (qwen Strict/Balanced/Lenient) |\nunanimous_pass → catch false |\nC Out-of-channel probe (task+artifact checksum; no judge rationale) |\n31/31 (100%)‡ |\n\n‡**Arm C is a fixture demonstration, not blind generalization.** `probe_fail()`\n\nencodes per-sid / task-constraint rules (including DS4); hold-out removes DS4 from the *recurrence registry only*. The probe still knows the artifact shape. Read 31/31 as “a checksum-style criterion *can* catch first occurrence without a class history entry,” not as “we discovered a probe without looking at the failure.” Same shape with `--holdout DS9`\n\n(re-run overwrites the JSON; default freeze is DS4).\n\nControls: recurrence on *non*-hold-out high-conf MISS = **61/61 (100%)** — history works when the class is known. Probe on legit V1/V2 = no false reject. Verdict: **SUPPORT** for the asymmetry shape (A misses novelty / B collapses / C can catch under oracle rules); **not** a claim that Arm C is production-ready or causally independent.\n\nSo Mike's fork is not only a definition. On this fixture: cheap recurrence misses never-burned classes; same-channel second reads collapse with the miss; a checksum-style probe *can* catch first occurrence without a registry entry — when you already know how to write the criterion. The novelty arm is still domain-shaped; that scarcity claim stays. Read with the Update above: this run supports the **structural** half of the novelty bar; it does not claim causal independence against same-pipeline common cause.\n\nPart 6 was right to stop majority-voting splits into a false consensus. It was wrong to treat the complement — unanimity — as safe auto-execute for the failure mode DF v2 already measured. Alexey named the population mismatch; the DF multi-perspective rerun puts numbers on it. Mike named the residual population the recurrence arm cannot see, pinned what “out-of-channel” must mean before that arm can be built, and split that bar into structural vs causal independence.\n\n**Divergence stays. T1/T2 join it. None of them is the novelty arm. A fifth prompt is not the novelty arm either. A checksum-passing “other data” probe is not automatically a common-cause shield.**\n\n**Series:** Agent Determinism Illusions · Scripts: [GitHub](https://github.com/zxpmail/blog/tree/main/agent-determinism-illusions/scripts)\n\n**Previous:** [Part 6 — Five comments that redesigned my LLM verification pipeline](https://dev.to/zxpmail/five-comments-that-redesigned-my-llm-verification-pipeline-388f)", "url": "https://wpnews.pro/news/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass", "canonical_source": "https://dev.to/zxpmail/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass-1513", "published_at": "2026-07-23 12:28:47+00:00", "updated_at": "2026-07-23 12:33:12.403796+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "large-language-models", "ai-agents"], "entities": ["Alexey Spinov", "Dipankar", "qwen3:0.5b", "deepseek DS4"], "alternates": {"html": "https://wpnews.pro/news/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass", "markdown": "https://wpnews.pro/news/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass.md", "text": "https://wpnews.pro/news/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass.txt", "jsonld": "https://wpnews.pro/news/divergence-escalates-the-wrong-population-unanimous-misses-auto-pass.jsonld"}}