# I picked a coding agent off a leaderboard. It flopped on our codebase. > Source: > Published: 2026-07-14 21:51:14+00:00 Last year my team had to pick a coding agent, and I volunteered to run the evaluation. I felt good about it. I pulled up the public benchmark scores, lined up the contenders, took the one at the top, and told everyone we had a winner. Then we actually pointed it at our repo. It did not blow up dramatically. It just kept being slightly wrong in ways that ate our time. It wrote diffs our reviewers would not approve. It renamed a function and broke three files it had never opened. The tests it ran passed, and the repo was still broken. I had confidently recommended a tool based on a number that turned out to say almost nothing about our situation. That was embarrassing enough that I went and figured out why. It took a few weeks of reading and a couple more bad calls before I landed on something that works. This is that, written plainly, and I hope it saves you the meeting where you have to walk your recommendation back. The score was not fake. It was just measuring somebody else's code. Once I looked properly, four gaps explained the whole thing: So I stopped treating the leaderboard as a decision and started treating it as a sniff test. Fine for narrowing the field. Useless for the actual call. Here is the bit I wish I had thought of on day one. The best possible test data for a coding agent is the work we had already merged. Our last fifty merged pull requests came with everything a benchmark cannot fake: the real intent, the reviewer's comments, a passing test suite, and a known-good final diff. So I replayed them: I got this wrong on my first attempt, so learn from it: I scored on how closely the text matched. That is both unfair and useless. A human reviewer would happily merge a differently-worded solution that behaves the same way. The question is not "does this look like our diff." It is "would this have passed the same review our diff passed." I also swap in a freshly merged PR every week and drop the oldest, otherwise the set slowly stops resembling what we ship. After running enough replays I stopped guessing and settled on five things. Each one catches something a benchmark never sees. **Would the diff pass our review?** Behavior, not text. I mark it down for reformatting files nobody asked it to touch, for breaking a test the merged version passed, and for taking twice the diff to reach the same outcome. **Did it use its tools in a sane order?** A coding agent is a loop, not a text generator. It reads, edits, runs tests, reacts, commits. I check four things: did it read a file before editing it, did the edit land where it said it would, did it actually run the tests afterward or just commit blind, and did it commit only what it changed. Editing a file it never opened is just guessing with confidence. **Did it update everything the change touched?** This is the one that burned us. The agent renames a helper in one file and leaves broken call sites scattered across others. The tests in the file it touched pass. The repo is wrecked. So now I follow the imports outward from every changed file and check that the downstream files still line up. **Did the plan survive the work?** Modern agents write a plan before touching anything, and that plan is worth grading. Two questions: did the plan match what was actually asked, and did the final diff still match the plan. Agents drift halfway through and quietly contradict the original request while the tests keep passing. This turned out to be the best predictor of which agent our reviewers ended up trusting, which I did not expect at all. **Does it know when to give up on a bad idea?** This was the sleeper, and it is now the one I care about most. When tests fail after an edit, a disciplined agent reverts and tries another approach. An undisciplined one piles more code on top of the failure until the suite is in pieces. You can see it plainly in the trace: how many edits does it make per failed test run, and does it ever actually revert. An agent averaging several edits per failure with no reverts will ship regressions. Count on it. The first three catch correctness. The fourth catches intent. The fifth catches whether you can trust it unsupervised. Coding agents invent credentials and echo risky shell patterns more often than I expected. So now every generated diff gets scanned for made-up secrets, dangerous constructs like raw `eval` or shell interpolation, and whatever patterns our own repo has banned. It takes milliseconds and it has already caught things. Cheap insurance. The agent that won our replay was not the one at the top of the leaderboard. It was the one whose plans stayed honest and that knew when to back out of a bad edit. Neither of those shows up in a public score, and both were exactly what our reviewers cared about. The line I settled on: the vendor owns the model and the prompt. I own the evaluation and the decision. And my evaluation is the only one that knows what our code actually looks like. If you have run something like this on your own repo, I would love to hear which dimension caught you off guard. For me it was rollback. It never occurred to me to measure whether an agent knows how to abandon a bad idea, and it turned out to matter more than almost anything else.