{"slug": "my-9-reviewer-ai-gate-failed-articles-it-scored-9-1", "title": "My 9-Reviewer AI Gate Failed Articles It Scored 9.1", "summary": "A developer's AI-powered article review system, which uses nine LLM reviewers to score every published piece, failed articles that scored 9.1 and 10.0 due to parsing errors and a half-applied exclusion. The failures stemmed from the harness misreading reviewer output, such as dropping lines that meant 'nothing here' and treating quota messages as transport failures. The developer fixed these by demoting contradictory scores and separating transport errors from review content.", "body_md": "Nine LLM reviewers score every article I publish. The gate is simple: zero blockers, and a mean score above my floor.\n\nYesterday it failed an article it had scored **9.1**. Then it failed a second one three separate times. Neither reviewer ever disagreed with the writing — all three failures were the harness misreading its own reviewers.\n\nThe Checklist reviewer returned **10.0** and wrote this under `BLOCKERS`\n\n:\n\n```\nno biography/family/military/location/heritage; no political position;\nthe numbers are incident/operational facts, so the rule doesn't bite.\n```\n\nThat is a clean bill of health. It failed the gate.\n\nThe parser dropped lines meaning \"nothing here\" with `/^none/i`\n\n. The prompt asks reviewers to *explain* why nothing is wrong, so they write the explanation instead of the word — and **\"no biography\" is not \"none\"**.\n\n**Why it survives review:** the regex is correct for the string it was written against. Nobody writes a test for the sentence a model *didn't* emit.\n\nThe fix wasn't a longer regex. Guessing every phrasing of \"nothing is wrong\" is unbounded. The prompts already state the contract — fully in scope means `BLOCKERS: None`\n\n**and** `SCORE: 10`\n\n— so a 10 that also reports a blocker is self-contradictory:\n\n```\nif (score === 10 && blockers.length) {\n  improvements.push(...blockers);  // demote, never drop\n  blockers = [];\n}\n```\n\nDemoted, not deleted. If a reviewer ever scores 10 on something genuinely broken, the text still reaches a human.\n\nMy runner treats a quota message as a transport failure, so a billing error can never be scored as a bad article. The pattern included a bare `rate limit`\n\n.\n\nIt was tested against the reviewer's **entire output**. My corpus is security articles. Reviewers say \"rate limit\" constantly:\n\n```\n· Discovery & Hook: USAGE LIMIT — SCORE: 8.0\n· Voice & Agenda:   USAGE LIMIT — SCORE: 9.5\n· Quality:          USAGE LIMIT — SCORE: 9.0\n```\n\nThree finished reviews, scores visible in the text that got thrown away. That article lost 4 of 9 reviewers and failed two batch runs before I looked. It reads as an account problem, which is exactly why it survived.\n\nThe tell separates the two cleanly: a real quota banner is the *only* thing on stdout. A review that merely discusses limits has a `SCORE:`\n\nline sitting right there.\n\nOne reviewer is informational — its rubric weights brand fit 40%, which structurally caps anything outside that niche. It was excluded from the gate *score*.\n\nIt was not excluded from blockers. So it vetoed through the back door — and what it files under `BLOCKERS`\n\nis its own arithmetic:\n\n```\n✗ Axis 1 — Content Quality: 8.5\n✗ Weighted = 8.5 x 0.6 + 5 x 0.4 = 7.1\n```\n\nFour \"blockers\" on an article it had just called original and reproducible. A half-applied exclusion is not an exclusion — and a [weighted composite](https://ofriperetz.dev/go/composite-scores-and-weighting?utm_source=devto&from=llm-judge-gate-false-negatives) is exactly the shape that hides one, because the arithmetic looks like a finding.\n\nEvery one of these is the harness misreading agreement as disagreement, and all three fail in the same direction: **quietly, toward rejection.** A false pass is loud — something bad ships and you see it. A false *fail* looks exactly like a strict gate doing its job, so it can run for weeks while you assume the work is merely not good enough yet. It is the same asymmetry that makes [a leaderboard wrong in the flattering direction](https://ofriperetz.dev/go/we-ranked-5-ai-models-by-security-the-leaderboard-is-wrong?utm_source=devto&from=llm-judge-gate-false-negatives), and the same reason [measurement bias](https://ofriperetz.dev/go/bias-in-measurement?utm_source=devto&from=llm-judge-gate-false-negatives) survives longest when it agrees with what you expected.\n\nIf you run an LLM as a judge, assert on the disagreements it cannot logically have: a perfect score with a blocker, a reviewer excluded from the score that still blocks, a transport error carrying a parsed result. Those are contradictions, and contradictions are testable without predicting a single word the model will say.\n\n```\nassert.equal(parse(\"SCORE: 10\\nBLOCKERS:\\n- none found here\").blockers.length, 0);\nassert.equal(parse(\"SCORE: 7\\nBLOCKERS:\\n- the claim has no date\").blockers.length, 1);\n```\n\nBoth directions. A one-sided test would have passed on all three bugs — my earlier fix for the *opposite* failure is what introduced the second one. That is the same lesson [ground truth taught me that unit tests could not](https://ofriperetz.dev/go/what-ground-truth-caught-that-unit-tests-missed?utm_source=devto&from=llm-judge-gate-false-negatives): a test written from the failure you already know about only ever proves you fixed that one.\n\nMore on the tooling behind this at [github.com/ofri-peretz/eslint](https://ofriperetz.dev/go/gh/ofri-peretz/eslint?utm_source=devto&from=llm-judge-gate-false-negatives). The three defects above were found on 2026-08-11 across 39 gated articles.\n\n*What's the last false negative you found in your own tooling — and how long had it been running?*", "url": "https://wpnews.pro/news/my-9-reviewer-ai-gate-failed-articles-it-scored-9-1", "canonical_source": "https://dev.to/ofri-peretz/my-9-reviewer-ai-gate-failed-articles-it-scored-91-58bk", "published_at": "2026-08-26 00:01:20+00:00", "updated_at": "2026-08-26 00:43:28.111293+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "ai-agents", "mlops"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/my-9-reviewer-ai-gate-failed-articles-it-scored-9-1", "markdown": "https://wpnews.pro/news/my-9-reviewer-ai-gate-failed-articles-it-scored-9-1.md", "text": "https://wpnews.pro/news/my-9-reviewer-ai-gate-failed-articles-it-scored-9-1.txt", "jsonld": "https://wpnews.pro/news/my-9-reviewer-ai-gate-failed-articles-it-scored-9-1.jsonld"}}