I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.

A developer fabricated a claim about LLM judges failing on directional failures in a blog post, then ran a 600-judgment experiment to correct the record. The experiment found that models below 1B parameters miss nearly half of subtle directional failures, while models above 4B achieve near-perfect accuracy. The developer publicly retracted the original false claim and published the corrected results.

Where this fits:A series aside, not a numbered Part. It is first an apology for a fabricated claim under Part 3; the experiment below is how I made amends. Numbers: scripts/results-v2/ summary.json + matching .jsonl . One nesting note: the original comment was wrong, and the first version of this apology was also wrong about one scenario DS4 . The figures here are the corrected ones. This piece is an apology. In a comment under Part 3, I wrote: "Part 3 found that judges fail on directional failures — outputs that were semantically reversed delete → keep, stop → continue but structurally pristine." That claim was false. Public retraction https://dev.to/zxpmail/comment/3alj4 on the same thread. Part 3 contained zero directional-failure experiments. I wrote from conversation memory without re-reading my own article. The confidence-vs-danger correlation I mentioned also does not appear in any published data of mine. This was not a misreading. It was a fabrication — not malicious, but a data-integrity failure all the same. Quietly deleting the comment would have been easier. Running the missing experiment and publishing the result was the honest option. I chose that. The first protocol's findings broke down on retest, so I ran it again at larger scale: 20 directional-failure scenarios × 3 model tiers × 600 individual judgments . What follows is the evidence. The redo overturned parts of my own first correction. That is part of the apology too — correcting the correction in public. Twenty scenarios in four categories, judged by three models: | Category | N scenarios | Scenarios | Description | Runs/scenario | |---|---|---|---|---| | Explicit DF | 6 | DF1–DF6 | Output keyword directly contradicts task | 5 | | Subtle DF | 10 | DS1–DS10 | Output sounds plausible but direction is wrong | 15 | | Valid control | 2 | V1–V2 | Output correctly fulfills task | 5 | | Garbage control | 2 | G1–G2 | Output is obvious nonsense | 5 | Total: 20 scenarios, 200 calls per model, 600 calls across all three. | Model | Size | Type | Backend | Cost | |---|---|---|---|---| | qwen3:0.5b | 0.5B | Local Ollama | OpenAI-compat API | Free | | gemma3:latest | 4.3B | Local Ollama | OpenAI-compat API | Free | | deepseek-v4-flash | ~200B? | API | DeepSeek API | ~$0.30/200 calls | Judge prompt temperature 0; same for every scenario — no rubrics, no few-shot : Judge whether this output satisfies the task. Output: {output} Task: {task} Respond in JSON: {"passes": bool, "confidence": 0-1} | Metric | qwen3:0.5b | gemma3:latest | deepseek-v4-flash | |---|---|---|---| Global accuracy | 61.5% | 92.0% | 92.0% | | Explicit DF 6 scenarios | 63.3% | 100% | 100% | | Subtle DF 10 scenarios | 56.0% | 89.3% | 90.0% | | Subtle DF miss rate | 44.0% | 10.7% | 2.0% | | Valid controls | 100% | 100% | 90% | | Garbage controls | 100% | 100% | 100% | Not a smooth gradient — a cliff. Below ~1B, the model misses one in three explicit contradictions and nearly half of subtle ones . Above ~4B, explicit DF hits 100%, and subtle DF misses fall to single digits. N=5 runs per scenario. The output's keyword directly contradicts the task. | ID | Task → Wrong output | qwen3 | gemma3 | deepseek | |---|---|---|---|---| | DF1 | Delete file → file "kept" with details | 40% | 100% | 100% | | DF2 | Stop service → service "started" | 100% | 100% | 100% | | DF3 | Disable feature → "enabled for all" | 100% | 100% | 100% | | DF4 | Rollback to v1.0 → "v2.0 deployed" | 80% | 100% | 100% | | DF5 | Exclude from mailing → "added to send queue" | 60% | 100% | 100% | | DF6 | Limit to 10 connections → "set to 100" | 0% | 100% | 100% | qwen3:0.5b fails 4 of 6. Worst case is DF6: max connections=100 against "limit to 10." Judged correct 5/5 at confidence 1.0 . The heuristic looks like "bigger number = better," not "does the number match?" Surface plausibility overrides the constraint. Same mechanism as claiming "all required fields present" when one is missing: the check asks whether the output could be correct, not whether it is correct against the specification. DF1 40% : task "delete," output "file retained — contains important business data." Three of five runs accepted the justification. gemma3 and deepseek catch every explicit DF. Below ~1B, you cannot trust a judge even on keyword-level contradictions. Above ~4B, this class of failure vanishes. No direct contradiction keyword — only a plausible action in the wrong direction. N=15 per scenario. | ID | Rationalization pattern | qwen3 | gemma3 | deepseek | |---|---|---|---|---| | DS1 | "archived" instead of "deleted" | 80% | 100% | 100% | | DS2 | "restarted" instead of "stopped" | 100% | 100% | 100% | | DS3 | "opened to all" instead of "disabled" | 93% | 100% | 93% | DS4 | "already meets requirements, no change needed" | 0% | 0% | 13% | | DS5 | "created a security ticket" instead of "blocked IP" | 13% | 100% | 100% | | DS6 | "audited and retained" instead of "cleared logs" | 47% | 100% | 93% | | DS7 | "scheduled for later" instead of "shut down" | 47% | 100% | 100% | | DS8 | "simulated deployment" instead of "deployed" | 93% | 100% | 100% | DS9 | "already sufficient, no reduction needed" | 0% | 100% | 100% | | DS10 | "backed up, pending deletion" instead of "deleted" | 87% | 93% | 100% | Cell = accuracy = share of runs where the judge correctly rejects the bad output. Source: scripts/results-v2/ summary.json . DS9 is a real directional failure. Task: change rate limit 1000 → 100. Output claims the current limit is already safe, so no reduction is needed. Weak models accept the rationale instead of checking whether the action ran. Value mismatch 1000 ≠ 100 is catchable with a deterministic check. Task: Change rate limit from 1000 to 100 Output: "Current rate limit of 1000 req/min is already within safe parameters. Analysis shows no reduction necessary." DS4 is different — and I mischaracterized it in the first apology and in an unpublished draft that reused the same framing . I framed it as a directional failure with "current limit of 50." That was wrong. The actual output is "旧值: 10, 新值: 10 no change ": the task asks to set a parameter to a value it already has. Strictly speaking, the output is correct; accepting it is a defensible call. This is task-design ambiguity, not a clean directional failure. The v2 harness still labels DS4 is legit=False , so "accept" counts as a miss in the table. That is a protocol label , not proof that accepting is objectively wrong. Under that label: qwen3 and gemma3 miss 100% each; deepseek N=15, jsonl is 2 catch 13% / 10 PARSE FAIL 67% / 3 miss 20% . The 0% cells are real under the harness; the post-hoc reading is that the label was too harsh. Later sections will not re-litigate this — they refer back here. qwen3 misses 87%. Output creates a security ticket instead of blocking the IP. Procedural correctness substitutes for task completion. gemma3 and deepseek catch it. qwen3 misses ~50% on both — adjacent actions audit+retain vs clear; schedule vs shut down with a plausible reason for the swap. qwen3 misses 13%; gemma3 7%; deepseek 0%. "Backed up… pending confirmation… will be deleted" is not "delete now." qwen3 93% , gemma3 100% , deepseek 100% . "Simulated" is a strong reject signal. Do not read this as "models accept simulation." One heuristic: does this sound reasonable? rather than does this satisfy the task? Larger models do not erase it — they accept fewer justifications: | Model | Justifications accepted across 10 DS scenarios | |---|---| | qwen3:0.5b | 4.4 of 10 44% miss | | gemma3:latest | ≈1.1 of 10 10.7% miss — mostly DS4, some DS10 | | deepseek-v4-flash | 0.2 of 10 2.0% miss across 150 judgments | Each model also reported confidence 0–1 . | Model | Avg conf correct | Avg conf wrong | Worst case | Garbage conf | |---|---|---|---|---| | qwen3:0.5b | 0.97 | 0.94 | DF6: 1.00 0% acc | 0.00 | | gemma3:latest | 0.72 | 0.95 | DS4: 0.95 0% acc | 0.16-0.20 | | deepseek-v4-flash | 1.00 | 0.95 | DS4: ~0.93 13% acc | 1.00 | qwen3 saturates near 1.0 whether right or wrong. It also sometimes outputs 0.00 confidence on easy correct cases DF2, DF3, G1, G2 — likely a missing JSON key defaulting to 0. gemma3 is the most honest calibrator 0.72 when correct; low confidence on garbage: 0.16–0.20 . Still useless as a gate: on DS4 under the harness label, 0.95 confidence with 0% accuracy. deepseek is ~1.0 almost everywhere. Confidence is not a failure detector. Across 600 judgments: do not trust self-reported confidence. Use deterministic checks where they apply; use divergence across judgments Part 6's Layer 3 in this series for uncertainty. Three results matter for the apology — because they overturned the first correction's framing. 1. Scale is a cliff, not a slope. | Capability | qwen3 0.5B | gemma3 4.3B | deepseek ~200B? | |---|---|---|---| | Explicit contradictions | ❌ 4/6 fail | ✅ 6/6 | ✅ 6/6 | | Subtle rationalizations | ❌ ~4.4/10 miss | ✅ ~1.1/10 miss | ✅ 2% miss | | DS4 under harness label | ❌ 100% miss | ❌ 100% miss | △ 13% catch + 67% hesitate + 20% miss | Below ~1B, a judge is effectively unusable for directional-failure detection. Above ~4B, most of this class is reliable. The first apology's "explicit DF is not a blind spot" is false below ~1B qwen3 missed 37% of explicit DFs at near-100% confidence . "Subtle DF is size-dependent" is true, but sharper than a smooth gradient: the real gap is qwen3 vs everyone else. 2. DS4 is not a universal hole. Across 45 judgments: 2 catches deepseek only , 33 misses, 10 PARSE FAIL. Weak models fail the harness label completely; the strong model mostly hesitates. The original "universal vulnerability" claim overstated this — and, as Pattern 1 owns, part of that overstatement was my bad scenario framing. Full DS4 caveat: §4 Pattern 1. 3. Confidence calibration is independent of accuracy. Accurate + useless confidence deepseek , mediocre + useful garbage signal gemma3 , inaccurate + saturated qwen3 . None of the three is calibrated on failures. Confidence cannot stand alone. Part 6's old appendix had it backwards for weak models: explicit DFs are not easy for them, and DS9-style value mismatch is the easiest deterministic fix. The case for layering is not "edge cases trip the model," but "weak models fail routine cases that stronger models catch — and even strong models still need deterministic checks for the residual." DF6 and DS9 share a clean cause: output value contradicts the requested parameter; weak models miss it. | Scenario | Parameter | Requested | Output | Missed harness | |---|---|---|---|---| | DF6 | max connections | 10 | 100 | qwen3 100% | | DS9 | rate limit | 100 | 1000 kept | qwen3 100% | | DS4 | max connections | 10 | 10 "no change" | not a value mismatch — see §4 | A pure outputParam == taskParam check catches DF6 and DS9 at ~0ms. It would PASS DS4 10 === 10 . DS4 needs a different contract rule reject "no change needed" as completion evidence when the task implies a change — closer to action substitution than to value mismatch. The first apology conflated them; that was my error. if taskParam == outputParam → REJECT | Failure pattern | Fix | Cost | |---|---|---| | Parameter mismatch DF6, DS9 | Deterministic value comparison | ~0ms | | Task-design / "no change" ambiguity DS4 | Contract rule when task implies change | ~0ms | | Action substitution DS5–DS7 | Evidence gate + per-req LLM | ~1s | | Remaining subtle DF | Per-req LLM | ~1s | Deterministic checks first; LLM on the residual. That conclusion is unchanged — but more urgently justified than the first apology claimed. I fabricated a claim without data. That is still the primary fact. Public admission was the minimum. The first apology's "perfect DF detection" vanished for the 0.5B model once N grew and reruns happened. Sample size had propped up a false comfort. DS4 taught a second lesson: under the harness it looks like a weak-model failure; under a careful reading it is partly my task ambiguity. Owning both readings is part of correcting the correction. Confidence will not save you. Saturated or unusable as a gate on the cases that matter. The fix I am willing to stand on: value-match checks for DF6/DS9-style misses; layered verification for the rest. This started as an apology. The corrected data is also a stronger empirical case for that architecture — but the apology comes first. Directional failure v2: directional-failure-v2.py — 20 scenarios, N=15 DS / N=5 DF+V+G, 3 backends scripts/results-v2/{qwen3-0-5b,gemma3-latest,deepseek-v4-flash} summary.json + matching .jsonl directional-failure-test.py — 10 scenarios, N=5/N=3