Heal Benchmark: AI Code Reviewers Miss 70% of Runtime Bugs, Claimed vs Verified

Heal, a vendor selling an agent that runs software to prove bugs with failing tests, published an October 2026 benchmark claiming that 70 percent of the user-facing defects its agent proved (141 of 202) were never flagged by CodeRabbit, Greptile or cubic across five products and 236 replayed pull requests. Heal's own figures show the miss rate ranged from 63 percent on Product A (a coaching app) and 58 percent on Product B (a contribution platform) to 87 percent on Formbricks, 94 percent on Sim and 100 percent on Infisical, while the review bots produced 1,003 distinct flags whose relevance Heal did not assess. Heal acknowledges the benchmark is vendor-run, with Heal choosing the products, method and language-model judge, and that the results were not independently reproduced.

AI code review has become standard on pull requests, and a new vendor benchmark argues it has a blind spot. Heal , a company selling an agent that runs your product and proves bugs with failing tests, published a benchmark https://www.heal.dev/benchmark?ref=explainx in October 2026 claiming that 70 percent of the user-facing defects its agent proved 141 of 202 were never flagged by CodeRabbit, Greptile or cubic . It also drew a Hacker News discussion, though a small one https://news.ycombinator.com/item?id=50258722&ref=explainx at the time of writing. The headline is catchy. The useful part is the fine print, which the page itself supplies. This post lays out the claim, what is and is not verified, and how to run a fairer test on your own repository. TL;DR: claimed vs verified | Claim | What the source shows | Status | |---|---|---| | 70% of Heal's defects were missed by review bots | 141 of 202 across five products | Heal's own figure; not independently reproduced | | Review bots find more issues overall | 1,003 distinct bot flags vs Heal's proven defects | Heal says so itself | | Stacking bots barely helps | Product A: about 24% caught by one bot, about 30% by three | Small sample, vendor judged | | Heal's defects are real | Each comes with a failing test | Stronger evidence than a comment | | Bot flags are real | Heal did not assess relevance | Unknown | How the benchmark worked Heal replayed each product's real pull request history, one PR at a time, into a fresh private repository with the three bots installed on default settings, and ran its agent before and after the sequence. Five products were used: two private codebases anonymised , and three open source projects, Formbricks https://github.com/formbricks/formbricks?ref=explainx , Sim and Infisical https://github.com/Infisical/infisical?ref=explainx . In total 236 PRs were replayed, with the open source ones covering short windows in mid October 2026. To decide whether a bot and Heal found the same thing, Heal used a language model as judge that "reads a bot's claim and a Heal defect on the same lines" and says whether they name the same problem. Heal says the judge was calibrated on a manually curated sample. Heal's agent works differently from a reviewer. It maps a product's entry points UI, API, CLI, SDK, webhooks , builds a sandbox that mocks or stubs dependencies, generates many candidate defects, and keeps only those it can reproduce with a failing test. A reviewer reads a diff and writes an argument; Heal tries to prove the bug. What the numbers say Per Heal's page, the share of Heal-proven user-facing defects that no review bot found was: | Product | Not found by code review | |---|---| | Product A coaching app | 63% | | Product B contribution platform | 58% | | Formbricks | 87% | | Sim | 94% | | Infisical | 100% | Heal gives examples it says a diff could not reveal: a frontend that builds old route paths after a backend rename, spread across two repositories inside one PR that only touched one of them; and an MFA flow where a failed organisation open after a correct code ends in a false lockout, requiring three separate pieces of code to be read together. That is a real class of problem. Bugs that emerge from interactions across files and services are hard to see in a diff, and running the software is a sensible way to find them. It matches the pattern we saw when AI-assisted code review shipped Postgres 19 bugs that had to be reverted https://explainx.ai/blog/postgres-19-delayed-ai-code-review-bugs-reverted-features-2026 . Where the claim is weaker Heal is unusually open about the limits, and they matter: 1. It is the vendor's benchmark. Heal chose the products, the method and the judge. 2. Relevance of bot flags was not assessed. The page says this "may create a bias in favor of code reviewers", but the comparison also only counts bug types Heal is built to find. 3. Small sample. Heal says it stopped "after spending several thousand dollars on code review" and calls the dataset relatively small. 4. Different scopes. Of the bots' 1,003 distinct flags, 654 were in files Heal does not read by design, such as CI configuration, migration history and analytics. Another 202 were on code Heal tests but did not report, mostly race conditions, and Heal did not check whether they were real bugs. 5. A failed first method. Heal says it first tried one-week windows of live pull requests and dropped them because attributing a defect to a single PR was unreliable. Replaying history fixed that, but replays removed any chance to see whether authors acted on bot comments. 6. Product A is early-stage. Heal notes its whole history was scanned, which favors both sides. None of that makes the finding false. It means 70 percent is best read as "in a vendor-built test, a large majority of runtime defects were outside what default-settings reviewers flagged," not as a general law. The stacking result Heal also argues that running several reviewers does not help much. On Product A, cubic alone caught 29 of the 120 defects Heal proved about 24 percent ; adding Greptile raised that to 35 and adding CodeRabbit to 36 about 30 percent . On Product B every combination caught 13 of 53 25 percent , on Formbricks 3 of 31, on Sim 1 of 21 and on Infisical none of 22. We read the page's chart this way; the labels are terse, so check the original figure. The takeaway Heal draws is that reviewers overlap heavily with each other. If that holds on your code, paying for three reviewers buys less than one reviewer plus a different kind of check. It is a useful hypothesis, and cheap to test. Why the gap is plausible, and why it is not the whole story A pull request reviewer sees a diff plus whatever context its index retrieves. That is excellent for style, obvious logic slips, missing null checks and security smells in the changed lines. It is structurally weaker at bugs whose cause is spread across files that the diff never touches together. Heal's backend-rename example is the textbook case: the diff in the backend repository is internally consistent, and the frontend that now builds dead URLs is not in the PR at all. Runtime testing has the opposite weakness. It only finds what it exercises. A single pass through a feature will not trigger a race condition reliably, which is why Heal concedes that most of the 202 flags it did not report were races. It also cannot say anything about code no user reaches: a leaked token in a CI workflow, a risky migration, or a dependency pinned to a vulnerable version. Those are exactly the categories that made up most of the bots' 1,003 flags. So the honest framing is not "reviewers are failing." It is that the two tools sample different parts of the bug space, and a team that relies on only one has a predictable gap. This is consistent with a theme we keep hitting in AI-assisted development: generating code is cheap, and verifying it is where the cost lands. Our look at vibe-coded agents and secure code https://explainx.ai/blog/last-secure-code-benchmark-vibe-coding-agents-fable-5-25-percent-secure-2026 reached a similar conclusion from the security side. Questions to ask any vendor benchmark The Heal page is better than many because it lists its limitations, but the same checklist applies to every benchmark in this space, including ones that favor reviewers: - Who ran it and who sells the winner? A vendor measuring itself against rivals will pick favorable ground. - Who judged equivalence? Here a language model decides whether two findings are the same. Judge errors in either direction change the percentage. - Were findings validated? Heal's defects come with failing tests; the bots' flags were not graded for relevance, so precision is unknown on one side. - What were the settings? Default configurations were used, and two of the open source repos shipped their own bot config files, which applies on those products. - How big is the sample and how recent? The open source windows were only a few days of PRs in October 2026. - Can you rerun it? The private products are anonymised, so only the open source portions could in principle be repeated. What to do with this You do not need Heal to act on the idea: - Keep your reviewer. The bot flagged categories Heal ignores, including CI risks. One flag in the page's own examples was a malicious PR injecting instructions into a CI assistant. - Add a runtime layer. End-to-end tests, a staging smoke run, or an agent that exercises the product catch cross-file mismatches. - Measure before buying more bots. Replay 30 recent merged PRs and count which bugs each tool would have caught. - Demand reproducibility. A finding with a failing test is cheaper to trust than a comment. For how real teams cut review risk, see how Claude Code's team review process uses risk tiers https://explainx.ai/blog/claude-code-team-code-review-process-ai-reviewer-risk-tiers-reverts-down-40-percent-2026 , and Alibaba's open code review CLI benchmark https://explainx.ai/blog/alibaba-open-code-review-ocr-cli-benchmark-2026 . To read vendor numbers like these critically, use our guide to AI benchmarks https://explainx.ai/blog/how-to-read-ai-benchmarks , and for the time cost of checking AI output see the verification tax https://explainx.ai/blog/ai-verification-tax-rework-sage-workday-foxit-surveys-how-to-cut-it-2026 . How to check the claim yourself 1. Pick 30 to 50 merged PRs from a repo where you know the bug history. 2. Run your review bots on a private copy of each PR on default settings. 3. Run an end-to-end suite or runtime agent on the final state. 4. Record only defects you can reproduce; label each as found by reviewer, by runtime, or both. 5. Compare overlap, noise false positives and cost per real bug. Our read The most credible part of the Heal page is its structure: reviewers and runtime testing find different things, and the overlap is small. The least credible part is the exact percentage, which depends on a vendor-chosen sample and a model judge. Treat 70 percent as a reason to run your own comparison, not as a measurement of the industry. Related reading Figures are from Heal's October 2026 benchmark page and may be updated by the vendor.