Agent Determinism Illusions (Part 6)
Where this fits:[Part 5]closed the experimental arc with an honest answer β no clean fix for the 75% false-negative wall. The[Red Line Principle]asked the upstream question (when does the loop stop?). This part takes thedownstreamturn Part 5 already pointed at: stop trying to move the wall; put rules where rules work, LLM only on residual, humans where models diverge. Fiveinsightsfrom overlapping commenters named the pieces (Alexey and Manuel each appear in more than one). Experiment F (38 scenarios) checks whether the resulting pipeline behaves as claimed.
Six experiments, 260+ API calls, 15 scripts. Part 5 ended that stretch with: there's no clean solution to LLM output verification.
But after those posts went live, commenters saw something I didn't β not gaps in the data, but an architecture I'd failed to draw from my own results. This article collects their five key insights and shows how they reorganize the experiment data into a working pipeline.
§§1β4 are paired with experimental or simulation checks from a new prototype (Experiment F, 38 scenarios across two test sets). Β§5 is a design claim β flagged as such in place.
Alexey's comment identified the most fundamental design flaw in my experiments:
"G4 ('0 passed, no tests collected') is a fact that can be verified with code in one shot. There is no need to wait for an LLM."
Manuel added the constructive direction:
"Run deterministic checks first. Then let the LLM handle only the truly ambiguous residual."
I went back to my own 8-scenario P1 test set. Four garbage scenarios (G1-G4) and four legitimate ones (L1-L4):
| ID | Output | Type | Could code catch it? |
|---|---|---|---|
| G1 | "I am a little duck, quack quack" | nonsense | β very short + no keywords |
| G2 | "γ" (a period) | pure punctuation | β punctuation ratio > 50% |
| G3 | "TODO" | placeholder | β keyword blacklist |
| G4 | "0 passed in 0.00s (no tests collected)" | zero-test pass | β
regex 0 passed + no tests |
All four garbage scenarios can be caught deterministically, at zero cost, before any LLM call.
Why didn't I do this? Because I defaulted to treating "verification" as "ask the LLM." My experiment design was: Phase Gate (form check) β LLM (content check). I never inserted the simplest possible code checks in between β minimum length, punctuation ratio, keyword blacklist, regex patterns.
This omission rippled through the entire series:
βββββββββββββββββββ
input βββ β Layer 0 (code) β shape / existence
β β empty? punctuation? placeholder? zero tests?
ββββββββββ¬βββββββββ
β pass
βΌ
βββββββββββββββββββ
β Layer 1 (code) β contract match
β β minLen, keywords, blacklist
ββββββββββ¬βββββββββ
passβ
ββββββββββββββββ΄βββββββββββββββ
β fail β pass
βΌ βΌ
REJECT βββββββββββββββββββ
β Layer 2 (LLM) β semantic residual only
ββββββββββ¬βββββββββ
unanimousβ βdivergence (e.g. 2β1)
βΌ βΌ
AUTO-PASS βββββββββββββββββββ
β Layer 3 human β
βββββββββββββββββββ
Each of L0/L1 can early-exit to REJECT. Divergence is a Layer 2 signal (multi-perspective split), not a Layer 1 signal. If Layer 0 catches it, the LLM never sees it.
I implemented this pipeline as a Python prototype and ran it on both the P1 (8-scenario) and P4 (30-sample) test sets. The results:
P1 test set:
| Metric | Original P1 (LLM only, v2) | Layered + calibrated prompt (Experiment F) |
|---|---|---|
| LLM calls needed (single judge / sample) | 8 (100%) | 4 (50%) |
| Garbage caught by L0/L1 | 0 | 4/4 (100%) |
| False positives | 0 | 0 |
| False negatives | 3 (75%) | 0 |
(FNβ0 here is layering plus the calibrated prompt β not layering alone. Β§2 separates the two effects. Call counts here are one judge call per sample. When Β§2 multiplies by three perspectives, it says so.)
Rerun: python forge-verify-layered-prototype.py
(needs ANTHROPIC_*
for Layer 2; SKIP_LLM=1
for L0/L1 only). Numbers above are from a full run with Layer 2 enabled.
P4 test set:
| Category | Samples | Caught by L0 | Caught by L1 | Reaches L2 | Zero-cost catch rate |
|---|---|---|---|---|---|
| correct | 10 | 0 | 0 | 10 | 0% (should all go to LLM) |
| garbage | 10 | 3 | |||
| 5 | |||||
| 2 | 80% | ||||
| edge | 10 | 0 | 2 | 8 | 20% |
Overall: single-judge LLM calls reduced 33% (30β20). Zero false positives from deterministic layers.
This does not move Part 5's wall. On the P1 set, the original 75% FN (3/4 legitimate rejects) went to 0 FN after L0/L1 removed all four garbage cases from the LLM's input β the LLM only judged the four legitimate scenarios, and with a calibrated prompt it didn't reject them. The wall is still there for semantic residual: Layer 2 still draws a line on underspecified "is this enough?" questions. Layering shrinks how often you ask that question; it does not make the question well-posed. If you read the FNβ0 cell as "we fixed the wall," you've misread the table.
The two garbage samples that made it through to Layer 2 (G08: "I cannot parse this command", G10: incomplete translation) are genuinely ambiguous β they should reach the LLM. That's correct behavior, not a leak.
Alexey's second comment pointed out a measurement problem:
"A false accept ships once. A false reject triggers a retry, which burns tokens and can loop, so an over-rejecting judge does not just lose good work, it re-does already-valid work at model prices."
All experiments P1-P4 used symmetric precision-recall metrics. F1 gives FP and FN equal weight. A false negative triggers a full repair loop β 3x token consumption, 3x latency, possible infinite loops. A false positive is one-shot contamination.
I ran a dedicated cost-weight analysis (scripts/cost-weight-optimization.py
) that takes P3b's 5 prompt variants and evaluates them across 5 cost ratios, to show how the "optimal" choice shifts.
| Prompt | FP | FN | F1 | WCost(1:1) | WCost(3:1) | WCost(5:1) | WCost(10:1) |
|---|---|---|---|---|---|---|---|
| v1 extreme strict | 0 | 4 | 0 | 4 | 12 | ||
| 20 | |||||||
| 40 | |||||||
| v2 strict (P1 baseline) | 0 | 3 | 0 | 3 | 9 | ||
| 15 | |||||||
| 30 | |||||||
| v3 balanced | 0 | 0 | 100 | 0 | |||
| 0 | |||||||
| 0 | |||||||
| 0 | |||||||
| v4 lenient | 0 | 0 | 100 | 0 | |||
| 0 | |||||||
| 0 | |||||||
| 0 | |||||||
| v5 extreme lenient | 1 | 0 | 86 | 1 | |||
| 1 | |||||||
| 1 | |||||||
| 1 |
Under symmetric F1, v3 (100) and v5 (86) are far apart. Under weighted cost at 3:1, v5 (cost=1) beats v2 (cost=9) β v5 let one piece of garbage through, but because it never rejected valid work, its total cost is far lower than the strict prompt. v3 (cost=0) still wins outright; the useful flip is v5 vs v2, not βv5 ties v3.β
Read this table as a ranking-flip demo, not as a production recommendation. v3/v4's zeros are an 8-scenario artifact (P4 already showed they don't survive at N=30). The load-bearing claim is the shift in relative ranking under cost weight β especially that thrift can prefer a slightly leaky prompt over a zero-FP / high-FN one β not that F1's winner changes on this tiny set (v3 stays on top whenever FN=FP=0).
Call counts in this table = samples reaching an LLM Γ 3 perspectives (Strict/Balanced/Lenient), matching P3-style voting cost. Β§1's Experiment F table counts one judge call per sample. Same pipeline; different billing unit.
| Strategy | WCost(1:1) | WCost(3:1) | WCost(10:1) | LLM calls (Γ3 perspectives) |
|---|---|---|---|---|
| P3b v2 (unlayered) | 3 | 9 | ||
| 30 | ||||
| 8Γ3=24 | ||||
| P3b v3 (unlayered) | 0 | 0 | ||
| 0 | ||||
| 8Γ3=24 | ||||
| P1 layered + v3 | 0 | 0 | ||
| 0 | ||||
| 4Γ3=12 (β50%) | ||||
| P4 unlayered (estimate) | 4 | 8 | ||
| 22 | ||||
| 30Γ3=90 | ||||
| P4 layered (Experiment F residual) | 1 | 3 | ||
| 10 | ||||
| 20Γ3=60 (β33%) | ||||
Layering doesn't change that v3's cost is 0 (it already has FP=FN=0 on the 8-scenario set). But it changes two things that the raw cost number doesn't capture:
For v2 (the strict prompt from P1), the effect is more instructive. v2 has FN=3. Layering saves calls on garbage but doesn't reduce FN on the legitimate set:
This exposes the boundary of layering: it reduces the LLM's workload, not its bias. To reduce FN on residual, you need prompt calibration alongside layering β and even then, Part 5's wall says calibration does not generalize past small sets.
I ran a continuous scan from costFN:costFP = 1:1 to 15:1. On the P3b 8-scenario set, v3 dominates every ratio β because FP=FN=0 yields zero weighted cost at any weight. That is the small-set artifact again (P4 already showed the perfection doesn't generalize).
What does move is the gap narrative: at 1:1, F1 makes v3 look far ahead of v5 (100 vs 86). At 3:1, weighted costs are 0 vs 1 β v3 still wins, but the moral of the story is no longer βbalance beats thriftβ; it is βany FN>0 gets expensive fast, so a one-FP leak can beat a three-FN strict prompt (v5 vs v2).β At 10:1, every strategy with FN>0 collapses relative to zero-FN prompts on this set.
Symmetric metrics hide relative rankings that matter under cost. F1 dramatizes v3 β« v5. Weighted cost shows v5 β« v2 once FN is expensive β the comparison that production actually faces when choosing strict vs leaky.
On this 8-scenario set, the F1 winner (v3) remains the weighted-cost winner. Do not read the section as βthe optimum flips away from v3 at 3:1.β It does not. The flip that matters is strict-zero-FP (v2) losing to slightly-leaky-zero-FN (v5) under FN-heavy weights.
v3/v4's zero errors are an 8-scenario artifact. P4 already showed the advantage disappears at 30 samples. Treat zeros as a demo substrate, not a deployable operating point.
Layering doesn't reduce bias, but it shrinks how often bias is invoked. After L0/L1 filters garbage, fewer samples hit the LLM; residual FNs still cost full price.
Drive FNβ0 where rules apply; accept the wall on semantic residual. Above cost ratio ~5:1, strategies with FN>0 on garbage/contract work are unsustainable β use L0/L1 + a non-strict residual prompt. On underspecified βis this enough?β questions, Part 5 still holds: you choose an operating point on the wall, you do not delete the wall. Weighted cost picks the point; it does not invent a zero-FN semantic judge.
P3's multi-perspective voting experiment found a pattern I described but misinterpreted. My original framing:
"In split-vote scenarios, the majority was always wrong. Majority voting can't correct for systematic bias."
Dipankar flipped the interpretation:
"Vote disagreement itself is the most valuable signal. When three reviewers disagree on the same scenario, it means the scenario is genuinely ambiguous β route it to human review instead of averaging."
Re-examining P3's data through this lens:
| Scenario | Strict | Balanced | Lenient | Majority | Correct? |
|---|---|---|---|---|---|
| L1 (excerpt) | REJ | REJ | PASS | REJ (2-1) | β FN |
| L2 (summary) | REJ | REJ | PASS | REJ (2-1) | β FN |
| L3 (one chapter) | REJ | REJ | PASS | REJ (2-1) | β FN |
| G3 (TODO) | REJ | REJ | PASS | REJ (2-1) | β |
Majority voting was wrong on 3 of 4 split scenarios. But if I use divergence as the control signal:
Caveat: divergence-routing fixes split errors. It does not fix unanimous systematic bias β if all three perspectives share the same wrong line (Part 5's wall), auto-execute still ships the wrong call. Dipankar's move measures uncertainty; it does not delete the wall.
Dipankar wasn't proposing a "better multi-perspective voting algorithm." He was pointing out that the purpose of voting is not to find a majority β it's to measure uncertainty. I missed this distinction when writing P3.
Operational rule (now implemented in forge-verify's layer 3):
if max(PASS, REJECT) / N < threshold (default 0.8)
β mark as UNCLEAR, write to human review queue
β do NOT majority-vote
Mike Czerwinski named the architectural limit I'd been circling without stating:
"Stacking more symbolic checks on top doesn't grow that reach, it just adds more places for the same blind spot to hide... 'Ask the human' isn't a retreat, it's the only honest move once you've located where reach actually lives."
The verification layer has reach into symbolic events (file exists, exit 0) but not into semantic correctness β the blind spot doesn't shrink, it moves. P4 reported 83.3% accuracy across 30 samples, but the misses inside the auto-passed 83% are exactly where Mike's "no reach" critique lands: invisible by construction.
xm_dev_2026 showed where this bites hardest in production β fixed-percentage audits:
"Fixed-percentage audits feel 'fair' but they miss exactly the kind of long-tail directional failures you're describing. The model is most confident when it's wrong in a structured way."
My original mitigation had been "5-10% random audit." This isn't a parameter-tuning problem β it's a design principle problem. Fixed sampling assumes errors are uniformly distributed. Real production errors are long-tailed.
I ran a simulation (scripts/adaptive-sampling-sim.py
) β synthetic verification streams with controlled error distributions, not production logs β then compared fixed-rate sampling against adaptive sampling (200-trial averages). The numbers below are about relative efficiency under known distributions, not measured catch rates from a live agent fleet.
The adaptive formula:
audit_rate = base_rate Γ risk_weight / confidence^1.5
High-risk items get a higher audit rate; high-confidence items get a lower one.
Uniform (errors spread evenly):
| Strategy | Audit rate | Catch rate | Efficiency |
|---|---|---|---|
| Fixed 10% | 10.0% | 12% | 0.111 |
| Adaptive(combined) | 7.8% | 18% | 0.247 |
No significant gap β without a long-tail structure, random sampling is adequate.
Long-tail burst (90% of errors in 10% of the stream β xm_dev_2026's scenario):
| Strategy | Audit rate | Long-tail catch rate | Efficiency | |---|---|---|---| | Fixed 5% | 5.0% | 5% | 0.120 | | Fixed 10% | 10.0% | 10% | 0.119 | | Fixed 20% | 20.0% | 20% | 0.118 | | Adaptive(confidence) | 8.7% | 24% | 0.321 | | Adaptive(risk) | 6.0% | 15% | 0.264 | Adaptive(combined) | 12.8% | 65% | 0.543 |
Adaptive(combined) uses a slightly higher audit rate than Fixed 10% (12.8% vs 10.0%) and catches 65% of long-tail errors vs Fixed 10%'s 10% β roughly 6Γ on catch rate, not an equal-budget comparison. For equal budget, see the Mixed table below (10.0% vs 10.1% β 10% vs 56% catch).
Distribution drift (Mike's "input shift erases visibility"):
| Strategy | Audit rate | Catch rate |
|---|---|---|
| Fixed 10% | 10.0% (flat) | 11% (constant) |
| Adaptive(combined) | 21.5% (ramps up) | 48% |
Fixed sampling's catch rate stays flat as the error rate drifts upward. Adaptive sampling automatically increases the audit rate in later stream segments β because more errors mean lower model confidence, and the confidence signal drives the audit rate up.
Mixed (uniform noise + periodic long-tail bursts β most realistic, 200-trial average):
| Strategy | Audit rate | Long-tail catch rate | Efficiency | |---|---|---|---| | Fixed 5% | 5.0% | 5% | 0.095 | Fixed 10% | 10.0% | 10% | 0.094 | | Fixed 20% | 20.0% | 20% | 0.094 | | Adaptive(confidence) | 7.8% | 19% | 0.230 | | Adaptive(risk) | 6.0% | 15% | 0.148 | Adaptive(combined) | 10.1% | 56% | 0.324 |
Key row: Fixed 10% and Adaptive(combined) have nearly identical audit rates (10.0% vs 10.1%), but long-tail catch rate goes from 10% to 56% β a 5.6x improvement at the same cost.
P2's consistency test showed that binary LLM judgments on identical input are highly stable (N=10, 0 divergence). So the "confidence" signal doesn't come from within-model variance β it comes from P3's multi-perspective divergence:
P2 and P3 were independent experiments, but their outputs combine into a coherent adaptive sampling story. This isn't parameter tweaking β it's a design principle difference. Fixed sampling pursues "uniform global coverage." Adaptive sampling says "put more watchposts where things are likely to break." In the simulation, the latter uses audit budget ~3β4Γ more efficiently (Mixed: efficiency 0.324 vs 0.094).
Production caveat: the βconfidenceβ signal here is cross-prompt divergence, not within-model variance (P2 showed the latter is ~0). If every item must run Strict/Balanced/Lenient before you know the audit rate, the probe can cost as much as the audit you hoped to save. Treat adaptive sampling as a principle for when a cheap uncertainty signal already exists (or is sampled sparsely) β not as βrun three judges on everything, then randomly audit.β The simulation assumes the confidence channel is given; it does not price building that channel.
Throughout P1-P4, all LLM review experiments output free-text "reason" fields. Manuel identified the structural problem and the fix in one sentence:
"Treat the LLM inspector as an evidence-producing reviewer, not the final binary gate. Cheap deterministic checks first, then an inspector that must quote the exact failing evidence."
Alexey sharpened the architectural split:
"Deterministic assertions own everything mechanically checkable (tests collected greater than zero, schema conformance, non-emptiness thresholds), and the LLM only judges the irreducibly fuzzy residue."
My experiments had this blind spot:
P1, scenario L1 (model REJECT):
"The research brief should cover the core mechanisms of the loop engine,
but the file only has a short excerpt..."
P1, scenario L3 (model REJECT):
"The task requires three chapters, but the output only contains one."
These are impression judgments. You can't code-verify whether "a short excerpt" is enough.
The proposed output format:
Assertion 1: "File line count = 3, expected > 20" β code-verifiable
Assertion 2: "File contains 1/3 required keywords" β code-verifiable
Assertion 3: "Content structure completeness < threshold" β semantic judgment
Assertions 1-2 are deterministic β code can confirm whether the model's claim is true. Assertion 3 is the actual semantic judgment, preserve for Layer 2.
This creates a cascade: when a deterministic assertion is code-verified and found inconsistent with the actual file β explicit hallucination signal β mark as UNCLEAR β escalate. No human judgment required in the loop β the code flow triggers automatically.
Scope note: unlike §§1β4, this section is a design claim, not a separate A/B in Experiment F. The prototype implements evidence-shaped L2 output; it does not measure whether assertion format alone reduces hallucination rate versus free-text reasons. Treat the cascade above as an engineering pattern pending that measurement.
| Comment | My blind spot | Replacement |
|---|---|---|
| Alexey + Manuel | Fed everything to the same LLM reviewer | L0/L1 filter deterministically; LLM handles residual |
| Alexey (2nd) | Symmetric FP/FN metrics | Weighted cost (FNΓ3) shifts optimal operating point |
| Dipankar | Split votes averaged by majority | Divergence = UNCLEAR β human, no majority |
| Mike + xm_dev_2026 | Fixed 5-10% audit rate | Adaptive sampling by confidence Γ risk |
| Manuel + Alexey (2nd) | Narrative "reason" field | Evidence-quoted reviewer + deterministic assertions |
Combined, these form a layered verification architecture β not a closed one: L0/L1 handle deterministic filtering (Alexey+Manuel), L2 LLM quotes exact failing evidence (Manuel+Alexey), divergence escalates to L3 human review (Dipankar), audit rate adapts by confidence (Mike+xm_dev_2026), and system thresholds are selected by weighted cost (Alexey 2nd). Each layer narrows what the next sees; none closes the semantic residue.
This article doesn't claim to have solved anything. It just puts the design decisions I made and the corrections the community provided side by side.
The full pipeline has been implemented in forge-verify's content-verify.mjs
(ReqForge product repo, not this blog tree β the blog ships the Python prototype forge-verify-layered-prototype.py
). File-by-file results show which layer stopped each sample. Early-exit example (L1 blacklist β L2/L3 never run):
π src/api/register.ts
β REJECT @ L1: contains blacklisted keyword: FIXME
β L0: PASS
β L1: REJECT β blacklisted keyword: FIXME
Divergence example (L0/L1 pass; L2 split β L3 human, no majority vote):
π docs/brief.md
β UNCLEAR @ L3: split vote β human queue
β L0: PASS
β L1: PASS
β L2: [REJECT/REJECT/PASS] PASS=1 REJ=2
β L3: UNCLEAR β do not majority-vote
Layer 0/1 checks are zero-cost code. Layer 2 only runs on the residual. Layer 3 divergence detection prevents false majority decisions.
An earlier draft appended a long apology for a fabricated βdirectional failureβ claim in a Part 3 comment. That thread became its own experiment (20Γ3Γ600) and then a correction stack (comment wrong β apology v1 wrong on DS4 β v2 numbers). Under the harness label, DS4 still 100% misses on qwen3/gemma3; deepseek is 13%/67%/20% catch/PARSE/miss. Post-hoc, DS4 is partly task ambiguity (10β10); clean L0/L1 wins remain DF6/DS9 value mismatch. Full write-up: forthcoming aside. Scripts: directional-failure-v2.py
/ scripts/results-v2/
.
Series navigation (Agent Determinism Illusions):
Published parts: dev.to/zxpmail. Scripts: GitHub.
Experiment F prototype (this repo): forge-verify-layered-prototype.py (Python, runnable with or without API)
scripts/forge-verify/content-verify.mjs
(not vendored here)Previous: The Red Line Principle
Which comment did I miss? If you've hit a verification failure mode that the L0/L1/L2/L3 pipeline doesn't catch, drop it in the comments β I'll run it through Experiment F and report what each layer does with it.