Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark

Shrijith Venkatramana, founder of Hexmos, argues that the AI Code Review Benchmark, while methodologically detailed, fails to address the fundamental problem of AI code review by defining the goal from first principles. He contends that the benchmark hits the wrong targets and that the industry is not yet ready for such a benchmark.

Benchmarks are interesting. They instantaneously give an air of something authoritative, objective, and precise, and for good reason. For most people, benchmarks look like something done by knowledgeable people on the topic—by those "who know better than me." However, until we go into the details of their exact workings, especially for those who are true students of a particular subject, relying on them blindly is a bad personal policy. If you actually know about a particular subject and care about it, I recommend you go deep into the reasoning of why particular metrics are worth your time and consideration, and try to make sense of things from first principles. In this case, I will go through the detailed Code Review Bench methodology https://github.com/withmartian/code-review-benchmark/blob/main/methodology/full.md and explain why I think tool builders in this category must look beyond the confines of the benchmark and think in a totally different way about the whole problem of AI code review. First and foremost, the AI Code Review Bench starts with seemingly good qualities: But for anyone who has gone into the depths of the problem and wrestled with the nitty-gritty of larger parts of the problem space, many shortcomings become apparent. Side Note: My Background I'm Shrijith Venkatramana, founder of Hexmos, presently building LiveReview , an org-wide harness for enforcing your engineering standards. My team and I have been building software systems together for the past 5 years in various capacities. I worked at Amazon before that, and studied Software Engineering under Prof. Crista Lopes at the University of California, Irvine. I've built engineering teams and systems for almost a decade. As someone with the background mentioned above, and since we do a lot of AI code reviews, issue detection, and so on, there were frequent requests from people asking, "How do you score on benchmark X or Y?" I'd casually answer that I am not entirely convinced by most benchmarks around, but felt like I ought to provide a deeper and detailed justification for my position. Hence, this post. In particular, the benchmark perfectly hits the wrong targets for what AI code review benchmarks should do. However, I don't mean to discourage or criticize the authors personally or anything of the sort: I believe they have put great effort into setting this system up, and I respect their efforts. I do believe they have made a truly honest effort in finding various potential issues with their work and tried to get as good a result as possible. Despite all that, I'd like to put my views on this benchmark, and benchmarks in general, on the record. First, the methodology makes an ambitious attempt to answer an important question: How should we benchmark AI code review? Unfortunately, I don't think the paper answers that question. In fact, I came away thinking it demonstrates why we're not yet ready for a benchmark. I argue that the methodology is excellent and detailed, but jumps to a solution before the problem of AI code review itself is defined from first principles. The paper openly acknowledges many major challenges in great detail: These are all highly valuable components in understanding what the "AI code review problem" is about. However, I still consider them lower priority issues. Some bigger, more important things have been totally sidestepped. To be specific, I believe that it misses the mark at the fundamental problem analysis stage. In the business world, Jeff Bezos articulated the "working backwards" process to build up Amazon. Although he was the first to institutionalize it, we find Steve Jobs working backwards from the customer experience before him. But then, why stop there? We can go back to George Polya, and in his How to Solve It , he specifies that step one is defining the goal, specifically the unknown, the x factor which needs to be unearthed. The problem solver must work backwards from the x to what they have. So, analysis is another word for the working backwards process. Once we have a good analysis, we can use what we have and turn it into what we want. The process of construction or moving forward based on the analysis is called synthesis . By the way, it was not Polya who came up with the idea of analysis and synthesis, but Pappus of Alexandria https://en.wikipedia.org/wiki/Pappus of Alexandria , an ancient mathematician of great mathematical insight and capability. Pappus was a big influence on Newton, who studied the analysis and synthesis methodology to get good at his scientific work. So, we go through all these references to Bezos, Jobs, Polya, and Pappus to mention the importance of defining the problem well . Interestingly, this methodology completely misses an attempt to characterize the problem of code review in explicit terms. That is, there is no organized effort to systematize: One can ask more questions of this sort to actually nail down the problem space to begin with. It looks like such a systematic inquiry was not performed by the benchmark authors. It just jumps into the solution space, which largely robs us of an opportunity to learn how the authors think about the problem in question. Despite them not clearly defining x , or the unknown, we still get a sense of how they look at the problem by piecing together the various statements made across the document. The first problem is that the benchmark largely treats AI code review as one problem. I think it has already become two. Humans have limited attention. The problem here is helping engineers understand what matters. Showing every possible issue is rarely optimal. Instead, information should be prioritized differently depending on: The problem is fundamentally one of recommendation rather than search. Success is measured by helping humans make better engineering decisions. The second problem has an entirely different consumer: another AI system. An agent does not become overwhelmed by hundreds or thousands of findings. In fact, exhaustive analysis is often desirable. The objective shifts toward reducing: Many findings never need to be shown to humans at all. They're simply passed to repair agents. The agent can handle volume; it doesn't get tired or overwhelmed. Volume and detail translate to higher capability for the repair agents. The paper acknowledges that autonomous agents will eventually require different evaluation methods. Where I disagree is that this is presented as a future transition. I think that future is already here. Many developers are no longer reading every review comment themselves. They're handing comments directly to coding agents, reviewing the resulting changes, and supervising the overall process. The workflow has already changed. That fundamentally changes what should be measured. One process optimizes for scarce human attention. The other optimizes for agent characteristics in the service of software quality. Once this distinction is made, many familiar benchmark concepts become much less obvious. Questions such as precision, recall, comment volume, and "noise" primarily belong to the human comprehension problem. Questions such as exhaustive verification, defensive hardening, and automated repair belong to the machine verification problem. The paper evaluates these under a single framework. I think separating them first would lead us toward very different benchmark designs. Metrics like comment volume, "noise", and precision have a very different meaning when another AI system—not a human—is consuming the output. To be fair, the document explicitly recognizes that human reviewers are imperfect and discusses multiple ways of eventually moving beyond human-generated gold sets. That is one of its strengths. However, the overall framing still begins with evaluating review performance against real-world human performance. I am not convinced that this is the right destination. Most code reviews happen under deadline pressure, reviewer fatigue, and incomplete context. Machines have obvious advantages: Humans contribute something different rather than simply being "better." Experienced engineers recognize architectural problems, product trade-offs, operational concerns, and business realities that exist outside the code itself. Machines and humans, therefore, have complementary strengths rather than existing on a single performance scale. The document rightly warns about Goodhart's Law. Ironically, I think it also demonstrates how difficult Goodhart's Law really is. The benchmark measures increasingly sophisticated proxies: Each of these is individually reasonable. None of them is actually the objective. The objective is producing better software. The further we optimize increasingly elaborate proxies, the easier it becomes to mistake them for the goal itself. Agreement with historical reviewers is not necessarily the same thing as reducing operational failures. One thing I genuinely appreciated about the paper is how candid it is. It openly discusses uncertainty around: What surprised me was that after acknowledging so many unresolved assumptions, the paper still culminates in a benchmark that inevitably appears objective. My concern is not that the benchmark is wrong. My concern is that the apparent precision of the benchmark exceeds our current understanding of the problem. Production bugs do appear in the methodology, but mostly as one signal among many. Personally, I would place operational outcomes much closer to the center. Many code review activities ultimately exist to reduce production failures, improve reliability, and avoid embarrassing incidents. Those outcomes deserve greater prominence than they currently receive. The paper spends considerable effort matching the observed distribution of historical bugs. I suspect severity is considerably more important. Different organizations legitimately tolerate different classes of issues. A regulated financial institution and an early-stage startup do not optimize for the same level of operational risk. Without organizational context, a universal benchmark inevitably averages together very different notions of software quality. Many valuable review comments concern things such as: These are difficult to benchmark because they extend beyond the code itself. Yet they are often exactly what distinguishes experienced reviewers. I also remain somewhat skeptical about using developer actions as evidence of comment quality. Increasingly, developers delegate review comments directly to coding agents. A comment being "acted upon" may therefore measure workflow automation as much as genuine human agreement. The paper already acknowledges several limitations of this proxy. My concern is simply that these behavioral signals may become progressively noisier as AI-assisted development becomes commonplace. If the distinction between human comprehension and machine verification is correct, I suspect future code review systems will evolve very differently. For humans, the problem becomes one of interface design. The important questions become: This looks much closer to a recommendation system than a traditional static analyzer. For AI agents, the optimization is almost the opposite: Find everything. Generate exhaustive evidence. Prioritize defensive hardening. Feed repair agents. Optimize for robustness rather than attention. Trying to evaluate both of these worlds using a single benchmark seems increasingly difficult. Despite my criticisms, I think this paper advances the discussion significantly. Among the ideas I found particularly valuable are: These are meaningful contributions. Even where I disagree with the conclusions, I found the discussion itself valuable. I came away thinking that the paper asks many of the right questions, but begins benchmarking before the underlying objective has fully stabilized. My disagreement is therefore less about the methodology than about the framing. I think AI code review is currently splitting into two different disciplines. One is about helping humans understand software. The other is about helping machines systematically harden software. Those are different problems, with different consumers, different interfaces, different optimization criteria, and ultimately different definitions of success. Until we separate them, I suspect any benchmark will necessarily measure a blended objective rather than a clearly defined one. For that reason, I see this paper not as the definitive benchmark for AI code review, but as an excellent exploration of why benchmarking AI code review remains an open research problem. AI-Assistance Disclosure: I spent in total 3 days to come up with this piece. I read through original materials myself, formulated all the key objections and conclusions myself and stand by the opinions expressed therein. At the same time, I have used AI to polish my writing at certain places, such as fixing punctuation, grammar, awkward phrases and faulty heading levels and so on. These are merely presentation and stylistic changes but nothing fundamental.