{"slug": "perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark", "title": "Perfectly Hitting the Wrong Target: The Story of an AI Code Review Benchmark", "summary": "Shrijith Venkatramana, founder of Hexmos, argues that the AI Code Review Benchmark, while methodologically detailed, fails to address the fundamental problem of AI code review by defining the goal from first principles. He contends that the benchmark hits the wrong targets and that the industry is not yet ready for such a benchmark.", "body_md": "Benchmarks are interesting. They instantaneously give an air of something authoritative, objective, and precise, and for good reason.\n\nFor most people, benchmarks look like something done by knowledgeable people on the topic—by those \"who know better than me.\"\n\nHowever, until we go into the details of their exact workings, especially for those who are true students of a particular subject, relying on them blindly is a bad personal policy.\n\nIf you actually know about a particular subject and care about it, I recommend you go deep into the reasoning of why particular metrics are worth your time and consideration, and try to make sense of things from first principles.\n\nIn this case, I will go through the detailed [Code Review Bench methodology](https://github.com/withmartian/code-review-benchmark/blob/main/methodology/full.md) and explain why I think tool builders in this category must look beyond the confines of the benchmark and think in a totally different way about the whole problem of AI code review.\n\nFirst and foremost, the AI Code Review Bench starts with seemingly good qualities:\n\nBut for anyone who has gone into the depths of the problem and wrestled with the nitty-gritty of larger parts of the problem space, many shortcomings become apparent.\n\n## Side Note: My Background\n\nI'm Shrijith Venkatramana, founder of Hexmos, presently building[LiveReview], an org-wide harness for enforcing your engineering standards. My team and I have been building software systems together for the past 5 years in various capacities. I worked at Amazon before that, and studied Software Engineering under Prof. Crista Lopes at the University of California, Irvine. I've built engineering teams and systems for almost a decade.\n\nAs someone with the background mentioned above, and since we do a lot of AI code reviews, issue detection, and so on, there were frequent requests from people asking, \"How do you score on benchmark X or Y?\" I'd casually answer that I am not entirely convinced by most benchmarks around, but felt like I ought to provide a deeper and detailed justification for my position. Hence, this post.\n\nIn particular, the benchmark perfectly hits the wrong targets for what AI code review benchmarks should do.\n\nHowever, I don't mean to discourage or criticize the authors personally or anything of the sort: I believe they have put great effort into setting this system up, and I respect their efforts. I do believe they have made a truly honest effort in finding various potential issues with their work and tried to get as good a result as possible.\n\nDespite all that, I'd like to put my views on this benchmark, and benchmarks in general, on the record.\n\nFirst, the methodology makes an ambitious attempt to answer an important question:\n\n**How should we benchmark AI code review?**\n\nUnfortunately, I don't think the paper answers that question.\n\nIn fact, I came away thinking it demonstrates why we're **not yet ready** for a benchmark.\n\nI argue that the methodology is excellent and detailed, but jumps to a solution before the problem of AI code review itself is defined from first principles.\n\nThe paper openly acknowledges many major challenges in great detail:\n\nThese are all highly valuable components in understanding what the \"AI code review problem\" is about.\n\nHowever, I still consider them *lower priority* issues. Some bigger, more important things have been totally sidestepped.\n\nTo be specific, I believe that it misses the mark at the fundamental problem analysis stage.\n\nIn the business world, Jeff Bezos articulated the \"working backwards\" process to build up Amazon. Although he was the first to institutionalize it, we find Steve Jobs working backwards from the customer experience before him.\n\nBut then, why stop there? We can go back to George Polya, and in his *How to Solve It*, he specifies that step one is defining the goal, specifically the unknown, the `x`\n\nfactor which needs to be unearthed. The problem solver must work backwards from the `x`\n\nto what they have. So, **analysis** is another word for the working backwards process. Once we have a good analysis, we can *use* what we have and turn it into what we want. The process of construction or moving forward based on the analysis is called **synthesis**.\n\nBy the way, it was not Polya who came up with the idea of analysis and synthesis, but [Pappus of Alexandria](https://en.wikipedia.org/wiki/Pappus_of_Alexandria), an ancient mathematician of great mathematical insight and capability. Pappus was a big influence on Newton, who studied the analysis and synthesis methodology to get good at his scientific work.\n\nSo, we go through all these references to Bezos, Jobs, Polya, and Pappus to mention the importance of **defining the problem well**. Interestingly, this methodology completely misses an attempt to characterize the problem of code review in explicit terms. That is, there is no organized effort to systematize:\n\nOne can ask more questions of this sort to actually nail down the problem space to begin with. It looks like such a systematic inquiry was not performed by the benchmark authors. It just jumps into the solution space, which largely robs us of an opportunity to learn how the authors think about the problem in question.\n\nDespite them not clearly defining `x`\n\n, or the unknown, we still get a sense of how they look at the problem by piecing together the various statements made across the document.\n\nThe first problem is that the benchmark largely treats AI code review as one problem.\n\nI think it has already become two.\n\nHumans have limited attention.\n\nThe problem here is helping engineers understand what matters.\n\nShowing every possible issue is rarely optimal. Instead, information should be prioritized differently depending on:\n\nThe problem is fundamentally one of recommendation rather than search.\n\nSuccess is measured by helping humans make better engineering decisions.\n\nThe second problem has an entirely different consumer: another AI system.\n\nAn agent does not become overwhelmed by hundreds or thousands of findings. In fact, exhaustive analysis is often desirable.\n\nThe objective shifts toward reducing:\n\nMany findings never need to be shown to humans at all. They're simply passed to repair agents.\n\n**The agent can handle volume; it doesn't get tired or overwhelmed.**\n\nVolume and detail translate to higher capability for the repair agents.\n\nThe paper acknowledges that autonomous agents will eventually require different evaluation methods.\n\nWhere I disagree is that this is presented as a future transition. I think that future is already here.\n\nMany developers are no longer reading every review comment themselves. They're handing comments directly to coding agents, reviewing the resulting changes, and supervising the overall process.\n\nThe workflow has already changed. That fundamentally changes what should be measured.\n\nOne process optimizes for scarce human attention.\n\nThe other optimizes for agent characteristics in the service of software quality.\n\nOnce this distinction is made, many familiar benchmark concepts become much less obvious.\n\nQuestions such as precision, recall, comment volume, and \"noise\" primarily belong to the human comprehension problem.\n\nQuestions such as exhaustive verification, defensive hardening, and automated repair belong to the machine verification problem.\n\nThe paper evaluates these under a single framework. I think separating them first would lead us toward very different benchmark designs.\n\nMetrics like comment volume, \"noise\", and precision have a very different meaning when another AI system—not a human—is consuming the output.\n\nTo be fair, the document explicitly recognizes that human reviewers are imperfect and discusses multiple ways of eventually moving beyond human-generated gold sets. That is one of its strengths.\n\nHowever, the overall framing still begins with evaluating review performance against real-world human performance.\n\nI am not convinced that this is the right destination.\n\nMost code reviews happen under deadline pressure, reviewer fatigue, and incomplete context.\n\nMachines have obvious advantages:\n\nHumans contribute something different rather than simply being \"better.\"\n\nExperienced engineers recognize architectural problems, product trade-offs, operational concerns, and business realities that exist outside the code itself.\n\nMachines and humans, therefore, have complementary strengths rather than existing on a single performance scale.\n\nThe document rightly warns about Goodhart's Law.\n\nIronically, I think it also demonstrates how difficult Goodhart's Law really is.\n\nThe benchmark measures increasingly sophisticated proxies:\n\nEach of these is individually reasonable. None of them is actually the objective.\n\nThe objective is producing better software.\n\nThe further we optimize increasingly elaborate proxies, the easier it becomes to mistake them for the goal itself.\n\nAgreement with historical reviewers is not necessarily the same thing as reducing operational failures.\n\nOne thing I genuinely appreciated about the paper is how candid it is.\n\nIt openly discusses uncertainty around:\n\nWhat surprised me was that after acknowledging so many unresolved assumptions, the paper still culminates in a benchmark that inevitably appears objective.\n\nMy concern is not that the benchmark is wrong.\n\nMy concern is that the apparent precision of the benchmark exceeds our current understanding of the problem.\n\nProduction bugs do appear in the methodology, but mostly as one signal among many.\n\nPersonally, I would place operational outcomes much closer to the center.\n\nMany code review activities ultimately exist to reduce production failures, improve reliability, and avoid embarrassing incidents.\n\nThose outcomes deserve greater prominence than they currently receive.\n\nThe paper spends considerable effort matching the observed distribution of historical bugs.\n\nI suspect severity is considerably more important.\n\nDifferent organizations legitimately tolerate different classes of issues.\n\nA regulated financial institution and an early-stage startup do not optimize for the same level of operational risk.\n\nWithout organizational context, a universal benchmark inevitably averages together very different notions of software quality.\n\nMany valuable review comments concern things such as:\n\nThese are difficult to benchmark because they extend beyond the code itself.\n\nYet they are often exactly what distinguishes experienced reviewers.\n\nI also remain somewhat skeptical about using developer actions as evidence of comment quality.\n\nIncreasingly, developers delegate review comments directly to coding agents.\n\nA comment being \"acted upon\" may therefore measure workflow automation as much as genuine human agreement.\n\nThe paper already acknowledges several limitations of this proxy.\n\nMy concern is simply that these behavioral signals may become progressively noisier as AI-assisted development becomes commonplace.\n\nIf the distinction between human comprehension and machine verification is correct, I suspect future code review systems will evolve very differently.\n\nFor humans, the problem becomes one of interface design.\n\nThe important questions become:\n\nThis looks much closer to a recommendation system than a traditional static analyzer.\n\nFor AI agents, the optimization is almost the opposite:\n\nFind everything. Generate exhaustive evidence. Prioritize defensive hardening. Feed repair agents. Optimize for robustness rather than attention.\n\nTrying to evaluate both of these worlds using a single benchmark seems increasingly difficult.\n\nDespite my criticisms, I think this paper advances the discussion significantly.\n\nAmong the ideas I found particularly valuable are:\n\nThese are meaningful contributions. Even where I disagree with the conclusions, I found the discussion itself valuable.\n\nI came away thinking that the paper asks many of the right questions, but begins benchmarking before the underlying objective has fully stabilized.\n\nMy disagreement is therefore less about the methodology than about the framing.\n\nI think AI code review is currently splitting into two different disciplines.\n\nOne is about helping humans understand software.\n\nThe other is about helping machines systematically harden software.\n\nThose are different problems, with different consumers, different interfaces, different optimization criteria, and ultimately different definitions of success.\n\nUntil we separate them, I suspect any benchmark will necessarily measure a blended objective rather than a clearly defined one.\n\nFor that reason, I see this paper not as the definitive benchmark for AI code review, but as an excellent exploration of why benchmarking AI code review remains an open research problem.\n\n*AI-Assistance Disclosure: I spent in total 3 days to come up with this piece. I read through original materials myself, formulated all the key objections and conclusions myself and stand by the opinions expressed therein. At the same time, I have used AI to polish my writing at certain places, such as fixing punctuation, grammar, awkward phrases and faulty heading levels and so on. These are merely presentation and stylistic changes but nothing fundamental.*", "url": "https://wpnews.pro/news/perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark", "canonical_source": "https://shrsv.hexmos.com/post/perfectly-hitting-the-wrong-target", "published_at": "2026-07-12 14:40:46+00:00", "updated_at": "2026-07-12 15:05:50.414633+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-tools", "ai-research", "developer-tools"], "entities": ["Shrijith Venkatramana", "Hexmos", "LiveReview", "Amazon", "University of California, Irvine", "Code Review Bench", "Jeff Bezos", "Steve Jobs"], "alternates": {"html": "https://wpnews.pro/news/perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark", "markdown": "https://wpnews.pro/news/perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark.md", "text": "https://wpnews.pro/news/perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark.txt", "jsonld": "https://wpnews.pro/news/perfectly-hitting-the-wrong-target-the-story-of-an-ai-code-review-benchmark.jsonld"}}