{"slug": "you-have-a-review-ceiling-measure-it-before-it-measures-you", "title": "You Have a Review Ceiling. Measure It Before It Measures You.", "summary": "A developer proposes measuring code review effectiveness by seeding known-defective changes, inspired by Margaret-Anne Storey's concept of cognitive debt. The method, akin to mutation testing, aims to detect when reviewers become rubber stamps as AI-generated pull requests increase.", "body_md": "I sat in on Margaret-Anne Storey's DORA community session last week, and she put a name on the thing I'd been circling since April. It isn't technical debt.\n\nHer [ACM Queue piece](https://queue.acm.org/detail.cfm?id=3807966) splits software health into three debts. Technical debt is the familiar one: implementation choices that make tomorrow's change harder. Intent debt is the missing rationale, the goals and constraints that say what a system is even for, which now has to be legible to agents and not just to people.\n\nCognitive debt is the one that stopped me. It's the erosion of shared understanding, the state where nobody on the team can confidently explain how the system works or predict what a change will break.\n\nRead that again if you review pull requests for a living.\n\nI closed [a thirteen-post retrospective](https://devopsdiary.blog/posts/thirteen-posts-looking-for-the-line/) last month admitting I couldn't answer one question: how many AI-generated pull requests a week can a review process absorb before it stops working as a control? I still don't have that number. What I have now is a name for what you accumulate while you don't have it, and a way to find yours.\n\nEvery metric most teams watch gets better as review collapses.\n\nMerge rate climbs. Time-to-approve drops. The throughput chart looks terrific right up until the incident review, because a reviewer who has quietly become a rubber stamp is indistinguishable from a fast reviewer in every dashboard you own today. Cognitive debt doesn't announce itself as a red number. It shows up as green ones, arriving faster.\n\nI know this failure mode from the inside. [Two months of green CI on conformance checks that had never once passed](https://devopsdiary.blog/posts/41-repos-in-five-months/), on my own project. A human audit caught it. No metric I was watching came close.\n\nWhat you need to measure is detection. Almost nobody does.\n\nWe solved this once already, for test suites.\n\nMutation testing injects known bugs into code and checks whether the tests catch them. A suite that passes everything might be thorough or might be asleep, and mutation testing is how you tell which. Nobody finds this exotic anymore.\n\nRun the same play against review.\n\nStart from a real baseline. [The delivery-layer metrics I'd instrument first](https://devopsdiary.blog/posts/two-layers-your-ai-sdlc-metrics-are-missing/) haven't changed since I wrote them down, so I won't re-list them here. Two weeks of honest numbers beats a quarter of arguing about which numbers to collect (the arguing is more fun, which is why teams pick it).\n\nThen seed canaries. Plant a small, steady stream of known-defective changes into the review queue under ordinary conditions. An off-by-one. A swallowed exception. A permission widened exactly one notch, the kind of diff that reads fine at 4:40 on a Thursday. Tell the team the program exists and never which PRs are in it, the way regulated shops have run surveillance testing against their own controls for decades.\n\nMore than instinct suggests. Detection rate is a proportion, so your error bars shrink with the square root of the count. Thirty canaries in a window gets you within 18 points at 95% confidence. A hundred narrows that to 10. Those are worst-case widths, and they tighten as detection climbs away from a coin flip.\n\nTen tells you nothing you didn't already believe.\n\nI'd start at one canary per twenty pull requests, measured in four-week windows. A team reviewing 200 PRs a week hits 40 canaries a month at that rate, which is enough to compare one window against the next. Seed heavier and the signal arrives sooner, paid for in reviewer hours spent on defects you planted yourself. Seed lighter and you'll still be waiting on your first usable number at Thanksgiving.\n\nBe honest about what that precision buys, because it isn't much. At 40 canaries a window you can see a collapse, detection dropping from 80% to 50%. You cannot see a 5-point drift, and anyone who tells you otherwise is selling something. This method finds cliffs. Gentle slopes stay invisible until they stop being gentle.\n\nThat's arithmetic. I haven't run it at a real team's volume, and the first person who does should publish what the numbers actually looked like.\n\nNow raise the load. As AI-generated PR volume climbs, detection against the canaries holds, then sags, then drops off a cliff, and the knee of that curve is your ceiling. That number is yours. It won't match anyone else's, and a vendor can't sell it to you.\n\n```\ndetection\nrate      *----*----*\n                     \\\n                      \\        <- the knee: your ceiling\n                       *\n                          *\n                              *\n          ---------------------------> AI PRs per week\n```\n\nPast the knee you aren't reviewing. You're taking on cognitive debt at a rate nobody is tracking, and you'll settle the balance during an outage.\n\nSeeding defects sounds dangerous. The canaries live behind a flag that never deploys, same as any dark launch.\n\nIt sounds like distrusting your engineers. You're auditing a process, which is what the word audit means (nobody reads a fire drill as an accusation).\n\nAnd yes, it's work nobody budgeted for. So was every control your shop now treats as table stakes, back when somebody first proposed it and got told there wasn't time.\n\nSet against that: you're already running the experiment. Every team that switched on AI-assisted coding is raising the load on its reviewers right now, today, with no curve and no instrument. The only open question is whether anyone is reading it.\n\nI'm building this into AIEOS so the answer is a query rather than a research project. One operator's numbers aren't an industry answer, though they're one more data point than anyone has published.\n\nSo measure it. Publish the curve, anonymized, methodology attached. Fifty teams publishing detection curves would teach this industry more about engineering in the AI era than every framework announcement of the last two years, mine included.", "url": "https://wpnews.pro/news/you-have-a-review-ceiling-measure-it-before-it-measures-you", "canonical_source": "https://dev.to/todd_linnertz_871a076f68e/you-have-a-review-ceiling-measure-it-before-it-measures-you-1lio", "published_at": "2026-09-03 03:14:07+00:00", "updated_at": "2026-09-03 03:22:43.367737+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents"], "entities": ["Margaret-Anne Storey", "DORA", "ACM Queue"], "alternates": {"html": "https://wpnews.pro/news/you-have-a-review-ceiling-measure-it-before-it-measures-you", "markdown": "https://wpnews.pro/news/you-have-a-review-ceiling-measure-it-before-it-measures-you.md", "text": "https://wpnews.pro/news/you-have-a-review-ceiling-measure-it-before-it-measures-you.txt", "jsonld": "https://wpnews.pro/news/you-have-a-review-ceiling-measure-it-before-it-measures-you.jsonld"}}