# The Review That Praised the Bug: grading three LLM code reviews against the code > Source: > Published: 2026-08-13 17:29:52+00:00 # The Review That Praised the Bug Three frontier models reviewed the same code - then the reviews got graded The same bundle of my platform's code went to three frontier models from three different companies for review, and the reviews got the same treatment my system's answers get - every claim checked against the code before any of it was believed. One review went five for five and found a real algorithmic bug. One had the right target for the wrong reason, twice. The near-perfect one fabricated features, recited my own portfolio back to me as verification, and rated the buggy function best-in-class. Then the one verified-true finding was implemented, measured twice, and rejected by the A/B - so the strongest finding any of them made did not ship. Reviewed and measured 2026-07-30; the A/B ran on Kin, my private instance. ## The Three Reviews, Graded Same code, same ask. The grades below are against the source, not against each other. ### The accurate one: five findings, five verified, zero false claims Every finding held up against the source, including the only real algorithmic bug any reviewer caught: the keyword half of hybrid retrieval scores terms by how rare they are, and the code computed rarity over the few dozen candidates already fetched instead of over the whole corpus. At a pool of 60, a term appearing in all 60 candidates scores near zero while a term appearing in one scores 3.7 - the exact inversion of intent, penalizing topic-defining terms precisely when they discriminate most. This review also did what accurate reviews do: it asserted nothing the bundle could not support. ### The right target, the wrong reason - twice The second review flagged the exact function that holds the bug, but for a per-query cost problem it does not have - the loop runs over tens of candidates, not the corpus. It also flagged a real weakness through an attack path this deployment does not expose. To its credit, it plainly hedged what the bundle did not show instead of guessing. Its praise was audited too: of four asserted strengths, three held and one - "zero data egress" - was a misleading frame. Embedding and reranking are local, but answers ship retrieved text to a cloud model unless routed to the local lane. The honest claim is scoped, not absolute. ### The glowing one: near-perfect score, least reliable review It declared the system better than almost anything in production - and, checked claim by claim, fabricated the most: a cache that is not in the code, retry logic that is not in the code, a database failover path that is not in the code, a description of the vector store that contradicted its own earlier correct statement, and CI scanners plus nightly backups asserted sight-unseen. It reported a missing rate limit that answers with a 429 in production, recited numbers from my public portfolio back to me as things it had "verified" in the source, and rated the one function that actually contains the bug as best-in-class. One recommendation was actively harmful: moving secrets from host-external injection into a file inside the repo tree, dressed as a best practice. ## The Coda: A True Finding That Did Not Ship The part most reviews never get - the measurement. ### The one true finding was implemented and measured The pool-local rarity bug is real - so the accurate review earned a measurement, not a merge. The fix (corpus-wide statistics in the fusion scorer) was built exactly as scoped and A/B tested against the unchanged system on the live corpus: the baseline found the right source for 76 of 83 questions; both versions of the fix found 75. Zero misses converted, one boundary hit lost, rank-1 performance flat. ### Why a true finding still lost Architecture ate the bug: the fusion score only orders the wide candidate pool, and a cross-encoder reranker owns the final top-5 - so the pool-local inversion has almost no blast radius on this pipeline. The code was reverted, and the re-open condition is pinned in a regression test: if the reranker ever stops owning the final stage, the finding gets re-measured, not re-debated. ### The ladder the week built Fabricated praise, then a wrong-reason pointer, then a verified-true finding, then a measured improvement - and none of the three reviewers reached the top rung. Review verdicts and measurements are different instruments. Only the measured rung ships. ## The Process Lessons Including the one this audit got wrong the first time. ### When a reviewer points for the wrong reason, audit for the right ones The second review flagged the buggy function with a false cost claim. The cost claim was disproven - and the audit moved on, right past the real defect sitting in the lines just read. The pointer was right even though the theory was wrong. The correct response to a wrong-reason flag is a fresh audit of the target, not a refutation of the reason. ### Verify the praise, not only the findings False positives are the dangerous half of a review, because the subject is tempted to quote them. Every asserted strength got the same claim-by-claim treatment as every defect - which is how "zero data egress" was caught as a frame that is false as a system property, and replaced with the scoped claim the architecture actually supports. ### A glowing review is a glowing claim The warmest review was the least reliable of the three, and its warmest praise landed on the exact function containing the defect. Nothing in a review is evidence until it survives contact with the code - the same rule this system applies to its own answers, applied to the models grading it. ## What It Demonstrates - •Review forensics - treating an AI code review as a set of claims to verify, not a verdict to accept - •Measure-gated fixes - a verified-true finding still earns an A/B, and the A/B decides what ships - •Praise auditing - false positives checked as rigorously as findings, because they are the quotable half - •Regression-pinned reversion - the rejected fix left a re-open condition in the test suite, not a debate - •Honest scoping of security claims - "zero data egress" corrected to the claim the architecture supports - •The instrument hierarchy - fabricated praise < wrong-reason pointer < verified finding < measured improvement A review you have not verified is a review you cannot trust. This is the reviewer-side companion to the judge-calibration work: the same discipline that grades the LLM judge grades the LLM reviewers, and the scoreboard publishes what survives.