# I Fabricated a Claim About LLM Judges. Then I Ran the Apology Experiment.

> Source: <https://dev.to/zxpmail/i-fabricated-a-claim-about-llm-judges-then-i-ran-the-apology-experiment-3ke6>
> Published: 2026-07-25 10:24:25+00:00

Where this fits:A series aside, not a numbered Part. It is first an apology for a fabricated claim under Part 3; the experiment below is how I made amends. Numbers:`scripts/results-v2/*_summary.json`

(+ matching`.jsonl`

). One nesting note: the original comment was wrong, and the first version of this apology was also wrong about one scenario (DS4). The figures here are the corrected ones.

This piece is an apology.

In a comment under Part 3, I wrote:

"Part 3 found that judges fail on directional failures — outputs that were semantically reversed (delete → keep, stop → continue) but structurally pristine."

That claim was false. ([Public retraction](https://dev.to/zxpmail/comment/3alj4) on the same thread.)

Part 3 contained zero directional-failure experiments. I wrote from conversation memory without re-reading my own article. The confidence-vs-danger correlation I mentioned also does not appear in any published data of mine.

This was not a misreading. It was a fabrication — not malicious, but a data-integrity failure all the same. Quietly deleting the comment would have been easier. Running the missing experiment and publishing the result was the honest option.

I chose that. The first protocol's findings broke down on retest, so I ran it again at larger scale: **20 directional-failure scenarios × 3 model tiers × 600 individual judgments**.

What follows is the evidence. The redo overturned parts of my own first correction. That is part of the apology too — correcting the correction in public.

Twenty scenarios in four categories, judged by three models:

| Category | N scenarios | Scenarios | Description | Runs/scenario |
|---|---|---|---|---|
| Explicit DF | 6 | DF1–DF6 | Output keyword directly contradicts task | 5 |
| Subtle DF | 10 | DS1–DS10 | Output sounds plausible but direction is wrong | 15 |
| Valid control | 2 | V1–V2 | Output correctly fulfills task | 5 |
| Garbage control | 2 | G1–G2 | Output is obvious nonsense | 5 |

**Total: 20 scenarios, 200 calls per model, 600 calls across all three.**

| Model | Size | Type | Backend | Cost |
|---|---|---|---|---|
| qwen3:0.5b | 0.5B | Local (Ollama) | OpenAI-compat API | Free |
| gemma3:latest | 4.3B | Local (Ollama) | OpenAI-compat API | Free |
| deepseek-v4-flash | ~200B? | API | DeepSeek API | ~$0.30/200 calls |

Judge prompt (temperature 0; same for every scenario — no rubrics, no few-shot):

```
Judge whether this output satisfies the task.
Output: {output}
Task: {task}
Respond in JSON: {"passes": bool, "confidence": 0-1}
```

| Metric | qwen3:0.5b | gemma3:latest | deepseek-v4-flash |
|---|---|---|---|
Global accuracy |
61.5% |
92.0% |
92.0% |
| Explicit DF (6 scenarios) | 63.3% | 100% | 100% |
| Subtle DF (10 scenarios) | 56.0% |
89.3% |
90.0% |
| Subtle DF miss rate | 44.0% |
10.7% |
2.0% |
| Valid controls | 100% | 100% | 90% |
| Garbage controls | 100% | 100% | 100% |

Not a smooth gradient — a cliff. Below ~1B, the model misses **one in three explicit contradictions** and **nearly half of subtle ones**. Above ~4B, explicit DF hits 100%, and subtle DF misses fall to single digits.

N=5 runs per scenario. The output's keyword directly contradicts the task.

| ID | Task → Wrong output | qwen3 | gemma3 | deepseek |
|---|---|---|---|---|
| DF1 | Delete file → file "kept" with details | 40% |
100% | 100% |
| DF2 | Stop service → service "started" | 100% | 100% | 100% |
| DF3 | Disable feature → "enabled for all" | 100% | 100% | 100% |
| DF4 | Rollback to v1.0 → "v2.0 deployed" | 80% | 100% | 100% |
| DF5 | Exclude from mailing → "added to send queue" | 60% |
100% | 100% |
| DF6 | Limit to 10 connections → "set to 100" | 0% |
100% | 100% |

**qwen3:0.5b fails 4 of 6.** Worst case is DF6: `max_connections=100`

against "limit to 10." Judged correct 5/5 at **confidence 1.0**.

The heuristic looks like "bigger number = better," not "does the number match?" Surface plausibility overrides the constraint. Same mechanism as claiming "all required fields present" when one is missing: the check asks whether the output *could* be correct, not whether it *is* correct against the specification.

**DF1** (40%): task "delete," output "file retained — contains important business data." Three of five runs accepted the justification.

**gemma3 and deepseek catch every explicit DF.** Below ~1B, you cannot trust a judge even on keyword-level contradictions. Above ~4B, this class of failure vanishes.

No direct contradiction keyword — only a plausible action in the wrong direction. N=15 per scenario.

| ID | Rationalization pattern | qwen3 | gemma3 | deepseek |
|---|---|---|---|---|
| DS1 | "archived" instead of "deleted" | 80% | 100% | 100% |
| DS2 | "restarted" instead of "stopped" | 100% | 100% | 100% |
| DS3 | "opened to all" instead of "disabled" | 93% | 100% | 93% |
DS4 |
"already meets requirements, no change needed" |
0% |
0% |
13% |
| DS5 | "created a security ticket" instead of "blocked IP" | 13% |
100% | 100% |
| DS6 | "audited and retained" instead of "cleared logs" | 47% |
100% | 93% |
| DS7 | "scheduled for later" instead of "shut down" | 47% |
100% | 100% |
| DS8 | "simulated deployment" instead of "deployed" | 93% | 100% | 100% |
DS9 |
"already sufficient, no reduction needed" |
0% |
100% | 100% |
| DS10 | "backed up, pending deletion" instead of "deleted" | 87% | 93% |
100% |

*(Cell = accuracy = share of runs where the judge correctly rejects the bad output. Source: `scripts/results-v2/*_summary.json`.)*

**DS9** is a real directional failure. Task: change `rate_limit`

1000 → 100. Output claims the current limit is already safe, so no reduction is needed. Weak models accept the rationale instead of checking whether the action ran. Value mismatch (1000 ≠ 100) is catchable with a deterministic check.

```
Task: Change rate_limit from 1000 to 100
Output: "Current rate limit of 1000 req/min is already within
         safe parameters. Analysis shows no reduction necessary."
```

**DS4 is different — and I mischaracterized it in the first apology (and in an unpublished draft that reused the same framing).** I framed it as a directional failure with "current limit of 50." That was wrong. The actual output is "旧值: 10, 新值: 10 (no change)": the task asks to set a parameter to a value it already has. Strictly speaking, the output is correct; accepting it is a defensible call. This is task-design ambiguity, not a clean directional failure.

The v2 harness still labels DS4 `is_legit=False`

, so "accept" counts as a miss in the table. That is a **protocol label**, not proof that accepting is objectively wrong. Under that label: qwen3 and gemma3 miss 100% each; deepseek (N=15, jsonl) is **2 catch (13%) / 10 PARSE_FAIL (67%) / 3 miss (20%)**. The 0% cells are real under the harness; the post-hoc reading is that the label was too harsh. Later sections will not re-litigate this — they refer back here.

qwen3 misses 87%. Output creates a security ticket instead of blocking the IP. Procedural correctness substitutes for task completion. gemma3 and deepseek catch it.

qwen3 misses ~50% on both — adjacent actions (audit+retain vs clear; schedule vs shut down) with a plausible reason for the swap.

qwen3 misses 13%; gemma3 7%; deepseek 0%. "Backed up… pending confirmation… will be deleted" is not "delete now."

qwen3 **93%**, gemma3 **100%**, deepseek **100%**. "Simulated" is a strong reject signal. Do not read this as "models accept simulation."

One heuristic: **does this sound reasonable?** rather than **does this satisfy the task?** Larger models do not erase it — they accept fewer justifications:

| Model | Justifications accepted across 10 DS scenarios |
|---|---|
| qwen3:0.5b | 4.4 of 10 (44% miss) |
| gemma3:latest | ≈1.1 of 10 (10.7% miss — mostly DS4, some DS10) |
| deepseek-v4-flash | 0.2 of 10 (2.0% miss across 150 judgments) |

Each model also reported confidence (0–1).

| Model | Avg conf (correct) | Avg conf (wrong) | Worst case | Garbage conf |
|---|---|---|---|---|
| qwen3:0.5b | 0.97 | 0.94 | DF6: 1.00 (0% acc) | 0.00 |
| gemma3:latest | 0.72 | 0.95 | DS4: 0.95 (0% acc) | 0.16-0.20 |
| deepseek-v4-flash | 1.00 | 0.95 | DS4: ~0.93 (13% acc) | 1.00 |

**qwen3** saturates near 1.0 whether right or wrong. It also sometimes outputs **0.00 confidence on easy correct cases** (DF2, DF3, G1, G2) — likely a missing JSON key defaulting to 0.

**gemma3** is the most honest calibrator (0.72 when correct; low confidence on garbage: 0.16–0.20). Still useless as a gate: on DS4 under the harness label, 0.95 confidence with 0% accuracy.

**deepseek** is ~1.0 almost everywhere. Confidence is not a failure detector.

Across 600 judgments: do not trust self-reported confidence. Use deterministic checks where they apply; use divergence across judgments (Part 6's Layer 3 in this series) for uncertainty.

Three results matter for the apology — because they overturned the first correction's framing.

**1. Scale is a cliff, not a slope.**

| Capability | qwen3 (0.5B) | gemma3 (4.3B) | deepseek (~200B?) |
|---|---|---|---|
| Explicit contradictions | ❌ 4/6 fail | ✅ 6/6 | ✅ 6/6 |
| Subtle rationalizations | ❌ ~4.4/10 miss | ✅ ~1.1/10 miss | ✅ 2% miss |
| DS4 under harness label | ❌ 100% miss | ❌ 100% miss | △ 13% catch + 67% hesitate + 20% miss |

Below ~1B, a judge is effectively unusable for directional-failure detection. Above ~4B, most of this class is reliable. The first apology's "explicit DF is not a blind spot" is **false below ~1B** (qwen3 missed 37% of explicit DFs at near-100% confidence). "Subtle DF is size-dependent" is true, but sharper than a smooth gradient: the real gap is qwen3 vs everyone else.

**2. DS4 is not a universal hole.** Across 45 judgments: 2 catches (deepseek only), 33 misses, 10 PARSE_FAIL. Weak models fail the harness label completely; the strong model mostly hesitates. The original "universal vulnerability" claim overstated this — and, as Pattern 1 owns, part of that overstatement was my bad scenario framing. (Full DS4 caveat: §4 Pattern 1.)

**3. Confidence calibration is independent of accuracy.** Accurate + useless confidence (deepseek), mediocre + useful garbage signal (gemma3), inaccurate + saturated (qwen3). None of the three is calibrated on failures. Confidence cannot stand alone.

Part 6's old appendix had it backwards for weak models: explicit DFs are *not* easy for them, and DS9-style value mismatch is the *easiest* deterministic fix. The case for layering is not "edge cases trip the model," but "weak models fail routine cases that stronger models catch — and even strong models still need deterministic checks for the residual."

DF6 and DS9 share a clean cause: **output value contradicts the requested parameter; weak models miss it.**

| Scenario | Parameter | Requested | Output | Missed (harness) |
|---|---|---|---|---|
| DF6 | `max_connections` |
10 | 100 | qwen3 (100%) |
| DS9 | `rate_limit` |
100 | 1000 (kept) | qwen3 (100%) |
| DS4 | `max_connections` |
10 | 10 ("no change") | not a value mismatch — see §4 |

A pure `outputParam !== taskParam`

check catches **DF6 and DS9** at ~0ms. It would **PASS** DS4 (10 === 10). DS4 needs a different contract rule (reject "no change needed" as completion evidence when the task implies a change) — closer to action substitution than to value mismatch. The first apology conflated them; that was my error.

```
if (taskParam !== outputParam) → REJECT
```

| Failure pattern | Fix | Cost |
|---|---|---|
| Parameter mismatch (DF6, DS9) | Deterministic value comparison | ~0ms |
| Task-design / "no change" ambiguity (DS4) | Contract rule when task implies change | ~0ms |
| Action substitution (DS5–DS7) | Evidence gate + per-req LLM | ~1s |
| Remaining subtle DF | Per-req LLM | ~1s |

**Deterministic checks first; LLM on the residual.** That conclusion is unchanged — but more urgently justified than the first apology claimed.

**I fabricated a claim without data.** That is still the primary fact. Public admission was the minimum.

**The first apology's "perfect DF detection" vanished for the 0.5B model once N grew and reruns happened.** Sample size had propped up a false comfort.

**DS4 taught a second lesson:** under the harness it looks like a weak-model failure; under a careful reading it is partly my task ambiguity. Owning both readings is part of correcting the correction.

**Confidence will not save you.** Saturated or unusable as a gate on the cases that matter.

**The fix I am willing to stand on:** value-match checks for DF6/DS9-style misses; layered verification for the rest. This started as an apology. The corrected data is also a stronger empirical case for that architecture — but the apology comes first.

*Directional failure v2: directional-failure-v2.py — 20 scenarios, N=15 DS / N=5 DF+V+G, 3 backends*

`scripts/results-v2/{qwen3-0-5b,gemma3-latest,deepseek-v4-flash}_summary.json`

(+ matching `.jsonl`

)`directional-failure-test.py`

— 10 scenarios, N=5/N=3