Every AI code review tool ships a blog post full of precision numbers. "98% precision, 87% recall, 40% fewer bugs shipped." I stopped trusting those the first time I ran one on a real repo and got 30 comments, 25 of which were pedantic or wrong.
Vendor benchmarks are evals the vendor picked, on repos the vendor picked, judged by a rubric the vendor wrote. Not useless. But not enough either. Here's the DIY method I run on my own repos before I decide whether a tool earns a seat in CI.
Take 20 merged PRs that already had real human review comments. Don't cherry-pick. You want a mix: ugly emergency fixes, long refactors, tiny one-liners. Run the tool against each PR's diff and classify every flag it produces into three buckets:
That's it. Don't score on a rubric. A human reviews the tool's output the way a human reviews a PR.
Two numbers matter way more than published precision:
Bad-advice rate (wrong / total flags). This is the dangerous one. A tool that flags a real race condition but also confidently suggests a fix that introduces a worse one has negative value, because your engineers will rubber-stamp it.
Triage burden (noise + wrong) / total. Every flag costs someone time deciding whether to act on it. If only 40% of flags are real, your senior dev does the tool's triage for it.
Running this on one repo took me an afternoon. It killed two products on my shortlist immediately and changed how I configure the one that survived.
Almost every violent miss was a settings or context problem, not a model problem. The tool that performed fine on the whole-repo context fell apart when pointed at a single file diff, because it couldn't see the calling code. The one that "over-reviewed" was tuned to flag anything, and the chase was breaking my team's thread. Defaults are tuned for marketing demos, not for your review culture.
Run the DIY eval once with defaults, then again with the tool pointed at your actual CI setup and your style of PR. The second run is the one that tells you anything.
Publishing a number on a self-chosen benchmark is easy. Doing the messy job of checking whether a tool helps your specific team, on your specific PRs, is the part worth your time.