I recently ran a small evaluation to compare three AI coding assistants. The task sounded straightforward: give each model the same engineering artifact, ask it to review the work, then score the results against a checklist I had prepared beforehand.
I expected to learn which model was the better reviewer. Instead, I learned something much more uncomfortable: my evaluation couldn't reward the behavior I actually wanted.
The checklist covered every category of issue I expected a competent reviewer to raise. Each finding earned points if it matched the checklist. That looked objective — until I realized what happened to findings that weren't on the list.
They automatically became false positives.
In other words, my evaluation couldn't possibly reward a model for discovering something I had missed. It only rewarded agreement with my own thinking.
I wasn't measuring "who reviews better." I was measuring "who thinks more like me." Those are not the same question.
At first I thought I simply needed a more complete checklist. That wasn't enough. No matter how much I expanded it, there would always be things I hadn't anticipated.
So I separated the results into two categories: expected findings, and new findings outside the checklist.
The important part wasn't creating the second column. It was refusing to judge it myself. If the entire purpose of that column is to capture things I failed to notice, I'm the least qualified person to decide whether those findings are valuable. So someone else reviewed those discoveries independently.
That changed the evaluation completely.
The new category produced two genuine issues that weren't on my original checklist. Both were things I had overlooked.
Even more interesting, each one came from a different model — and neither model surfaced the other's finding. If I had picked a single "winner," one of those two issues would never have shown up.
That made me rethink the question itself. Instead of asking "Which reviewer is the best?" I started asking "Which reviewers complement each other?" Those are very different evaluation goals.
Another observation was just as interesting. When the task was "follow explicit instructions and revise the work," the models performed similarly. When the task became "find something nobody explicitly asked you to find," the differences became much larger.
The ranking wasn't universal. It depended on the role.
That sounds obvious in hindsight, but many comparisons quietly assume a model has a single global ranking across every engineering task. My experiment suggested otherwise. Choosing an editor and choosing a reviewer may be two entirely different optimization problems.
I still use checklists. They're valuable because they make evaluations repeatable. But now I treat them as only half of the measurement. The other half is explicitly looking for discoveries that the checklist couldn't have predicted.
Otherwise, an evaluation becomes surprisingly good at rewarding agreement — and surprisingly bad at finding the thing you actually wanted someone to notice.
Disclosure: I used AI assistance to improve the English wording of this article. The engineering experience, analysis, and conclusions are my own.