{"slug": "counting-bugs-is-the-hard-part-of-comparing-ai-review-tools", "title": "Counting bugs is the hard part of comparing AI review tools", "summary": "An analysis of the Entelligence benchmark comparing GPT-5.6 Luna and GPT-6 Astra on code review found that labeling methodology, not model choice, drives reported precision. Run on 2026-09-14 against 50 public PRs, Luna reported 69 verified bugs at 74% precision for $0.20, while Astra reported 92 at 96% precision for $5.66, with two judges agreeing on 91% of findings. The piece argues that precision and recall figures are unverifiable without a disclosed label protocol, judge independence, and disagreement rates.", "body_md": "Model benchmarks for code review usually report the same two numbers: how many bugs the tool found, and how many of its comments were actually right. The part nobody can verify is the counting. Who decided a finding was a real bug? That single choice moves precision by more than the model choice does.\n\nThe Entelligence benchmark comparing GPT-5.6 Luna and GPT-6 Astra on review is a useful case, because it is one of the few vendors that publishes its method. Ran on 2026-09-14, it put both models against 50 public PRs from the AI-Code-Review-Evals dataset (Cal.com, Sentry, Discourse, Keycloak, Grafana).\n\nAcross those PRs Luna reported 69 verified bugs and Astra 92. Luna's precision was 74%, Astra's 96%. Total review cost was $0.20 for Luna and $5.66 for Astra. The headline is that a model roughly 28 times cheaper misses fewer than a third more bugs. The part worth staring at is the verification.\n\nEntelligence had two judges score every finding, and the two agreed on 91% of them. A bug only counted when both judges called it real. That is the right shape of a method, and it is rare in the category. Most comparison posts that reach you quote a recall figure with no statement of who labeled the ground truth, whether the labeler worked for the vendor, or how two judges resolved disagreement.\n\nHere is why the counting matters more than the model. A correctness label has a huge surface: a finding can be a true bug, a style nit, a misunderstanding of the code, or a duplicate of something a human already flagged. Where a vendor draws those lines determines precision. The same model run could get 80% or 95% precision depending on whether the checker counts near-duplicates as hits. So \"96% precision\" without the label protocol is not a number you can act on. It is a claim[reference:[https://entelligence.ai/blogs/gpt-5.6-luna-vs-gpt-6-astra-is-a-1.20-model-good-enough-for-code-review](https://entelligence.ai/blogs/gpt-5.6-luna-vs-gpt-6-astra-is-a-1.20-model-good-enough-for-code-review)].\n\nThe practical consequence for a team: when you read a tool comparison, ask for three things before trusting a precision or recall figure. The label protocol out loud, not a link buried in an appendix. Whether the judge who built the product also scored it, since a vendor judging its own output is a conflict, even an honest one. And the disagreement rate between judges, because 91% agreement means 9% of the hard calls went the scoring team's way, and those are exactly the security-critical findings you care about.\n\nNone of this tells you which tool to buy, and it does not tell Entelligence's models what to fix when precision drops. On Keycloak's auth code, Luna's precision fell toward 50%, meaning a working review on permission logic was flipping a coin on whether a comment was real[reference:[https://agentwrotethis.dev/blog/multi-repo-ai-review-a-context-problem-not-a-volume-problem/](https://agentwrotethis.dev/blog/multi-repo-ai-review-a-context-problem-not-a-volume-problem/)]. The counter for that is not a better benchmark. It is a verify step a human owns.\n\nThe takeaway is operational: treat a review-tool recall figure as unverified until you know who called the ground truth. Vendor benchmarks in this category publish method so rarely that the ones that do are worth reading twice, and the ones that do not are worth ignoring until they do.", "url": "https://wpnews.pro/news/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools", "canonical_source": "https://dev.to/tessainsley/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools-1cm1", "published_at": "2026-09-16 00:15:10+00:00", "updated_at": "2026-09-16 00:37:43.285643+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-research"], "entities": ["Entelligence", "GPT-5.6 Luna", "GPT-6 Astra", "Cal.com", "Sentry", "Discourse", "Keycloak", "Grafana"], "alternates": {"html": "https://wpnews.pro/news/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools", "markdown": "https://wpnews.pro/news/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools.md", "text": "https://wpnews.pro/news/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools.txt", "jsonld": "https://wpnews.pro/news/counting-bugs-is-the-hard-part-of-comparing-ai-review-tools.jsonld"}}