cd /news/ai-tools/a-code-review-benchmark-that-isn-t-t… · home topics ai-tools article
[ARTICLE · art-134853] src=dev.to ↗ pub= topic=ai-tools verified=true sentiment=· neutral

A code review benchmark that isn't the vendor ranking itself

AI research lab Martian has launched Code Review Bench, an open, MIT-licensed benchmark that scores AI code review tools on precision, recall, and F1 across 16,017 real open source pull requests where developers actually acted on bot suggestions. The leaderboard ranks 14 tools, including GitHub Copilot, CodeRabbit, Greptile, Qodo, Cursor, Claude, and Devin, and shows no single dominant tool, with Greptile posting the best precision but the worst recall among leaders and CodeAnt reaching only 38 percent recall. Martian positions the benchmark as a neutral alternative to vendor-published rankings that place their own products first.

by read5 min views2 publishedSep 20, 2026

Ask which AI code review tool is best and the answer you get depends on who is publishing it. The deepsource.com listicle ranks CodeRabbit first and runs on a code-quality product. codeant.ai's own 2026 roundup rates CodeAnt AI the best overall, in a list it wrote, after advertising "300K-PR". The monterail.com comparison is an agency, not a neutral source. Every ranking in that SERP has an owner.

So the useful answer to "what are the best AI code review tools" is not another list. It is a scoreboard nobody paid for, with the method published next to the numbers. One now exists.

Martian, an AI research lab rather than a code review tool vendor, runs Code Review Bench. The methodology is public: it pulled real open source pull requests on GitHub where AI review bots actively participated, then reconstructed each review timeline, from bot suggestion to developer response to the code change that actually landed. A suggestion counts as a hit only when a developer acted on it. Each tool gets a precision, a recall, and an F1 score, and the whole thing is reproducible because the code-review-benchmark repo is MIT-licensed on GitHub.

That last detail is the one that separates it from every vendor listicle. The benchmark is not a PDF a company commissioned to sell seats. It is code you can run, and the input set is described, not hidden behind a login. When a number has no method attached, you have a claim, not a result. This one publishes its method.

The live leaderboard scored 16,017 pull requests. On the overall F1 ranking, the spread at the top is narrower than the marketed "best" language suggests:

Read the precision and recall columns before you read the F1. The number to extract is how a tool trades thoroughness against noise. Greptile scores the best precision, which means when it comments, developers act on it most of the time. It also scores the worst recall among the leaders, so it leaves the most issues unreported. CodeAnt sits at the opposite extreme of this sample: precise comments, but a recall of 38 percent means it misses most of what a reviewer would flag. The "balanced" tools, Copilot and Coderabbit, sit near the middle of both.

Sample sizes are part of the method and the comparison. GitHub Copilot was scored across 767 PRs here, CodeRabbit across 2,109, Qodo across 2,208. A short sample on a leaderboard is a warning sign, not a headline. Prefer a tool whose position survives its sample size.

The second thing to notice is who is ranked at all. The leaderboard runs 14 tools, from the platform-native Copilot and Cursor to standalone reviewers like CodeRabbit, Greptile, and Qodo, to coding agents whose review output got scored too, like Claude and Devin. Between precisions of 80.3 and 63, there is no single tool that dominates, and the gap between the top F1 and the bottom is roughly 15 points.

That is the honest shape of this market. No vendor's marketing copy will show you the other 13 rows. A benchmark run by a neutral party does, which is why the tool ranked last on it is still liable to cite it when the numbers are on its side elsewhere. When a vendor ranks itself first in its own list, that is not evidence. When a tool you do not sponsor finishes ahead of the field on an open benchmark, that is at least a starting point.

A benchmark tells you which tools are worth a trial, not which one to buy. Use it to thin the candidate set, then apply the filters no F1 score captures.

Start with where the data comes from. The benchmark scores open source pull requests, weighted on whether a developer adopted the tool's suggestion. Your closed codebase, compliance rules, and review culture are not that sample. A high recall tool will look different on a team that already ignores half its bot comments than on one that routes everything through automated review.

Then fold in the dimensions a leaderboard cannot rank: platform fit, per-seat cost against volume, and whether you need self-hosting. Pricing pages are the primary source for those, and they move monthly. Greptile prices by credits and seats, CodeRabbit sells on a 14-day trial with no permanent free tier, Qodo has spun free-tiers for OSS and a commercial SaaS. Tools built to sit inside a team already running GitHub-native review, like Kodus, trade raw bug-recall for a handoff a human can actually process in the triage time allotted. None of that shows up in an F1.

The practical move is to take the two or three tools whose precision and recall look right for your noise tolerance, run them on a defined slice of your real pull requests for two weeks, and count the suggestions your own reviewers would have made. That count is the only benchmark that is about your code. The public leaderboard gets you to that trial faster, and it does it without asking you to trust the people selling you the answer.

Before this, "best AI code review tool" routed you to whichever company wrote the ranking. That is still most of the top results, which is why comparisons of review tools keep coming back to the same self-reported numbers with no method attached. An independent, open, reproducible scoreboard does not settle which tool is right for you, but it removes the layer where the vendor controls the evidence. Treat the 14-row table as the floor for the conversation, not the last word.

── more in #ai-tools 4 stories · sorted by recency
── more on @martian 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/a-code-review-benchm…] indexed:0 read:5min 2026-09-20 ·