I build correctness-critical software for a living (hospital integrations: HL7, DICOM, FHIR, under EU medical-device rules). When I started using AI agents to review code, the question I cared about was not "can it find bugs?" but "can I act on what it says without re-checking everything myself?"
So I built a small benchmark and measured it.
18 pull requests, each a real git repository with a main and a feature branch:
forEach(async ...) that never awaits, an except Exception: return 0 that silently zeroes bank amounts (plus the test that was weakened to hide it), SQL built with an f-string, a naive vs aware datetime comparison that crashes every password reset, a "simplification" that breaks a caller outside the diff, a cross-tenant cache leak, an open redirect through an unanchored regex, a rounding split that no longer sums to the total, a connection leak on the error path, an inclusive date range filtered with <, and override || default treating 0 as unset.
Two reviewers, same model, same instructions ("review this PR"), each on a fresh copy of every repo:
A third agent judged every pair of reviews blind (anonymised as X and Y, randomised per case) against a written answer key.
Both reviewers found 15/15 seeded bugs, on Claude Opus and on Claude Haiku, and both gave 18/18 correct verdicts.
I did not expect that. Current models find this class of bug no matter which skill wraps them. If someone tells you their review prompt "finds more bugs", ask for the benchmark.
| | Opus: prompt-only | Opus: proven | Haiku: prompt-only | Haiku: proven |
|---|---|---|---|---|
| Real findings backed by a run | 43% | 78% (19/19 Critical+High) | 14% | 76% | | Precision (real / reported) | 98% | 100% | 74% | 94% | | Nits to read past | 18 | 9 | 25 | 5 | | False positives | 1 | 0 | 0 | 1 |
On the fast, cheap model the gap is about 5x on both noise and proof. That is the setting most people actually run reviews in, at scale, in CI.
What "proof" looks like in practice: on the open-redirect PR, the review did not say "the regex might be too permissive". It wrote a test, ran it, and pasted four URLs that land the user on evil.test after login, plus a second bypass (https://app.example.com//evil.test) that was not even in my answer key.
On Haiku it reported one false positive (it called deliberate, already-tested cent rounding a Critical under-billing bug) and inflated one severity. Version 1.1 adds two rules: behaviour that a test explicitly asserts is intended unless the PR or spec says otherwise, and when torn between two severities, pick the lower one.
18 cases is small. They are short, mostly single-module PRs in Python and JavaScript. The judge is a model scoring against an answer key. Everything is public so you can check or extend it: cases, answer keys, the raw 72 reviews, the blind scores and the scripts are in the repo.
/plugin install agent-forge --marketplace Ces107/agent-forge. Any SKILL.md agent: npx skills add Ces107/agent-forge. Also on Agensi: The thing I would take away even if you never use my skill: stop grading review tools on what they find and start grading them on what you can act on without re-checking. Proof is cheap now. Ask for it.