The Bug Was in the Grader

A developer found that bugs in evaluation systems, not models, were the root cause of poor performance across three different AI projects. In one case, a flawed exam caused all models, including GPT-5.5, to fail equally. In another, a mismatched evaluation gate blocked a denial-management engine despite strong performance. The developer emphasizes that evaluation code must be treated as critically as model code.

The most expensive bug I shipped last month was not in a model, and it was not in a prompt. It was in the thing doing the grading. I found it three times, in three different businesses, in the same month. Each time I had been blaming the wrong thing. I run a small training platform. Five open-source models from the Qwen and DeepSeek families, one hard 54-task exam, and a gate that only promotes a model when the numbers say the win is real. For weeks none of them could get through the gate. The easy explanation was the comfortable one: small models, hard exam, keep training. So I plugged a frontier model, gpt-5.5, into the exact same harness, just as a reference. Same tasks. Same fixtures. Same scoring. It scored 0.241. That is the same as all five of the small models. The paired McNemar test put p at 0.31 or higher on every comparison, which is a careful way of saying there was no real difference between any of them. And 34 of the 54 tasks were failed by every model, the frontier one included. There are only two ways to read a number like that. Either frontier models are a lie, or the exam is broken. The exam was broken. Evidence from one task was leaking into unrelated tasks, and a bug in the harness was quietly cutting off the reference model's answers. What I wrote in the log that day was simple: the thing holding us back is the quality of the tasks and fixtures, not the models. I had spent all my effort on the students. The bug was in the professor. Key insight:When every model fails the same questions, the problem is the exam, not the models. Your eval is code too, and its bugs are the worst kind, because they decide what "passing" even means. Same month, different business. My denial-management engine reads healthcare claim denials and picks the move: appeal, fix and resubmit, bill the patient, or write it off. Tuning the prompts against a golden set took mapped-action accuracy from 36.7% to 76.7%, and classification from 80% to 93.3%, with hard errors down to zero. It was getting sharp. The ship gate was not. It required the tuned system to agree with a weaker, raw agent, and it reported that they agreed only 44.7% of the time. That looked alarming. It was also wrong. When I read through the 150-case run by hand, 27 of the "disagreements" were things like appeal level 1 versus peer-to-peer versus retro authorization. All the same kind of move. Things a human biller files in the same drawer. And the bug underneath was almost embarrassing. One axis of the eval compared raw label strings, and the other axis used a map that knew which labels meant the same thing. One eval, two rulers. Measured the same way on both sides, and against the golden answers instead of a weaker sibling, the tuned system sat at 92.0% classification and 90.7% action. On the cases where it really did disagree with golden, it still matched about 78% of the time. The agent managed 38%. I threw out the old gate and set a new one against golden truth at 90%. The old gate, in my own words from the log, measured the wrong thing and could never pass. Making your best rater agree with your worst one is not rigor. It is a ceiling you built yourself. Third business, same lesson, turned inside out. My Medicare remote-monitoring compliance platform lives in a world where the grader is not mine to fix. Federal contractors run screens against billing patterns, and the government publishes exactly what those screens are. So the platform runs the auditor's own math on itself. Every month it computes those screens over its own claims and writes the results down as snapshots that cannot be edited. A correction is a new version, never a quiet overwrite. Even a screen that does not apply gets written down as N/A, with a reason, so an auditor can see it was considered and not just skipped. The whole goal fits in one sentence: sound the alarm before the outside contractor would. That is what grading the grader looks like when the grader outranks you. Run their test yourself. Run it early. Keep the receipts. Three businesses. A training loop for open models, a denial engine, a Medicare compliance platform. In all three, the best engineering I did that month was not on the system being measured. It was on the thing doing the measuring. Your model is probably fine. Your prompt is probably fine. Go read your eval line by line, because the most expensive bug you have right now is the one that quietly decides what passing means. None of this is theory. It is a list of things that actually happened in one month. If you have ever caught your own grader lying to you, you know the feeling. Originally published at nabbilkhan.com.