My 9-Reviewer AI Gate Failed Articles It Scored 9.1

A developer's AI-powered article review system, which uses nine LLM reviewers to score every published piece, failed articles that scored 9.1 and 10.0 due to parsing errors and a half-applied exclusion. The failures stemmed from the harness misreading reviewer output, such as dropping lines that meant 'nothing here' and treating quota messages as transport failures. The developer fixed these by demoting contradictory scores and separating transport errors from review content.

Nine LLM reviewers score every article I publish. The gate is simple: zero blockers, and a mean score above my floor. Yesterday it failed an article it had scored 9.1 . Then it failed a second one three separate times. Neither reviewer ever disagreed with the writing — all three failures were the harness misreading its own reviewers. The Checklist reviewer returned 10.0 and wrote this under BLOCKERS : no biography/family/military/location/heritage; no political position; the numbers are incident/operational facts, so the rule doesn't bite. That is a clean bill of health. It failed the gate. The parser dropped lines meaning "nothing here" with /^none/i . The prompt asks reviewers to explain why nothing is wrong, so they write the explanation instead of the word — and "no biography" is not "none" . Why it survives review: the regex is correct for the string it was written against. Nobody writes a test for the sentence a model didn't emit. The fix wasn't a longer regex. Guessing every phrasing of "nothing is wrong" is unbounded. The prompts already state the contract — fully in scope means BLOCKERS: None and SCORE: 10 — so a 10 that also reports a blocker is self-contradictory: if score === 10 && blockers.length { improvements.push ...blockers ; // demote, never drop blockers = ; } Demoted, not deleted. If a reviewer ever scores 10 on something genuinely broken, the text still reaches a human. My runner treats a quota message as a transport failure, so a billing error can never be scored as a bad article. The pattern included a bare rate limit . It was tested against the reviewer's entire output . My corpus is security articles. Reviewers say "rate limit" constantly: · Discovery & Hook: USAGE LIMIT — SCORE: 8.0 · Voice & Agenda: USAGE LIMIT — SCORE: 9.5 · Quality: USAGE LIMIT — SCORE: 9.0 Three finished reviews, scores visible in the text that got thrown away. That article lost 4 of 9 reviewers and failed two batch runs before I looked. It reads as an account problem, which is exactly why it survived. The tell separates the two cleanly: a real quota banner is the only thing on stdout. A review that merely discusses limits has a SCORE: line sitting right there. One reviewer is informational — its rubric weights brand fit 40%, which structurally caps anything outside that niche. It was excluded from the gate score . It was not excluded from blockers. So it vetoed through the back door — and what it files under BLOCKERS is its own arithmetic: ✗ Axis 1 — Content Quality: 8.5 ✗ Weighted = 8.5 x 0.6 + 5 x 0.4 = 7.1 Four "blockers" on an article it had just called original and reproducible. A half-applied exclusion is not an exclusion — and a weighted composite https://ofriperetz.dev/go/composite-scores-and-weighting?utm source=devto&from=llm-judge-gate-false-negatives is exactly the shape that hides one, because the arithmetic looks like a finding. Every one of these is the harness misreading agreement as disagreement, and all three fail in the same direction: quietly, toward rejection. A false pass is loud — something bad ships and you see it. A false fail looks exactly like a strict gate doing its job, so it can run for weeks while you assume the work is merely not good enough yet. It is the same asymmetry that makes a leaderboard wrong in the flattering direction https://ofriperetz.dev/go/we-ranked-5-ai-models-by-security-the-leaderboard-is-wrong?utm source=devto&from=llm-judge-gate-false-negatives , and the same reason measurement bias https://ofriperetz.dev/go/bias-in-measurement?utm source=devto&from=llm-judge-gate-false-negatives survives longest when it agrees with what you expected. If you run an LLM as a judge, assert on the disagreements it cannot logically have: a perfect score with a blocker, a reviewer excluded from the score that still blocks, a transport error carrying a parsed result. Those are contradictions, and contradictions are testable without predicting a single word the model will say. assert.equal parse "SCORE: 10\nBLOCKERS:\n- none found here" .blockers.length, 0 ; assert.equal parse "SCORE: 7\nBLOCKERS:\n- the claim has no date" .blockers.length, 1 ; Both directions. A one-sided test would have passed on all three bugs — my earlier fix for the opposite failure is what introduced the second one. That is the same lesson ground truth taught me that unit tests could not https://ofriperetz.dev/go/what-ground-truth-caught-that-unit-tests-missed?utm source=devto&from=llm-judge-gate-false-negatives : a test written from the failure you already know about only ever proves you fixed that one. More on the tooling behind this at github.com/ofri-peretz/eslint https://ofriperetz.dev/go/gh/ofri-peretz/eslint?utm source=devto&from=llm-judge-gate-false-negatives . The three defects above were found on 2026-08-11 across 39 gated articles. What's the last false negative you found in your own tooling — and how long had it been running?