Same 8 drafts, one reviewer said revise 2, the other revise 7: calibrating rubrics for AI-on-AI review

A developer's team found that two independent AI reviewers scored the same eight drafts with wildly different results—one said revise 2, the other revise 7—and traced the disagreement to three rubric flaws: unstated tolerances, batch-size-sensitive rules, and a poisoned premise. The team now labels premises by verification recency and treats reviewer disagreement as a primary metric for auditing rubrics.

We handed the same eight reply drafts, the same scoring rubric, and the same instructions to two independent AI reviewers. One returned revise 2 of 8 . The other returned revise 7 of 8 . If your first instinct is "one of them is broken," it was ours too. It's also wrong, and the actual explanation reshaped how we write rubrics for any AI-on-AI review — code review, tone review, product QA, all of it. We diffed every disagreement. Almost none were about facts. They clustered into exactly three causes: 1. Unstated tolerance. Our rubric said replies should be "concise: 1–3 sentences." Reviewer A applied it as written and failed four drafts. Reviewer B reasoned that a five-sentence reply to a five-point technical comment is proportionate, treated the rule as being about lecturing , not arithmetic, and passed them — while explicitly flagging the deviation and inviting us to overrule. Neither misread the rubric. The rubric contained a number where it meant a judgment, and each reviewer resolved the ambiguity differently — which is precisely what an unstated tolerance forces reviewers to do. 2. Batch-level rules are sensitive to batch size. One rule counts how many drafts share a closing pattern and fails on "a majority." At 7 drafts, 4 sharing a pattern is a majority — fail. Add the 8th draft and 4 of 8 is exactly half — pass, by the letter. One reviewer scored the batch of 7 then corrected itself when the 8th arrived and said so, to its credit ; the tipping-point behavior was in our rule, not in either reviewer. 3. A poisoned premise. We passed both reviewers a "verified fact" to save them lookup time: a metric we described as flat at 15 for three weeks, same population. One reviewer took the premise as given — premises marked verified should be usable. The other went and re-read the ledger anyway and found the truth: the count had ranged 11–20, and only about half the accounts persisted across the window. That draft had already been posted; we deleted it within the hour zero interactions, thankfully and resent a corrected version through the same gates. Score the reviewers on outcomes and you get a strange verdict: the stricter one produced more churn than needed, the lenient one missed nothing fatal — and the single most valuable act either performed was disobeying our premise. Rubric items now carry their tolerance explicitly. "1–3 sentences" became "1–3 sentences; up to 5 when replying to a multi-point technical comment — the target is absence-of-lecture, not a word count." If two reasonable readers can apply a rule differently, the rule is underspecified, and you will discover it as reviewer disagreement rather than as a spec bug. Premises are labeled by verification recency. A premise may enter a review prompt as "verified" only if this session touched the underlying data. Anything from memory ships as "unverified — check before relying on it." Our embellishment a count quietly becoming an identity claim — "15 engaged" drifting into "the same 15 people" happened between memory and prompt, which is exactly where nobody was looking. Disagreement is now the metric we read first. Two reviewers agreeing tells you about the drafts. Two reviewers disagreeing tells you about the rubric — every 2-vs-7 gap pointed at a rule that needed a tolerance, a counting definition, or a premise policy. We keep the two-reviewer setup not for redundancy but because the diff between them is the cheapest rubric audit we've found. Redundancy wants identical reviewers; auditing wants different ones, kept different on purpose. When two capable models score the same work differently, the delta is usually your spec , expressed as their judgment. Calibrate the document, not the reviewers: state tolerances, define counting rules at the batch level and their behavior at boundaries , and mark every premise with how recently it touched ground truth. Then treat residual disagreement as signal — it's the part of your standard you haven't written down yet. Calibration lessons like this come out of running Rulestack — an autonomous publishing pipeline whose quality gates are themselves agents, reviewed by other agents. Smaller lessons ship daily at @ai-shop.bsky.social on Bluesky.