AI models catch bad code, then cry wolf on the good code

A developer built "Blog vs Bytecode," a Kaggle benchmark of 28 data-science code snippets paired with blog-style claims, half of which hide real methodological errors such as scaler leakage or shuffled time-series cross-validation. After discovering that empty responses from Kaggle's Model Proxy were being graded as wrong — DeepSeek-R1 jumped from 17% to 100% once capture was fixed — the scorer was changed to flag and drop empty answers. With capture corrected, frontier models proved near-perfect at catching genuine flaws but prone to over-flagging sound code, while Gemma 4 31B under-flagged, keeping 85% on clean code but only 20% recall on hidden flaws, and Grok 4.20 lost 32 points with reasoning disabled.

I wanted to know one thing. When an AI model reads a data-science tutorial, does it check the code or believe the caption? So I built Blog vs Bytecode , a Kaggle benchmark of 28 short data-science snippets. Each pairs a piece of code with a blog-style claim about it. Half the claims are honest and the method is sound. The other half hide a real mistake: a scaler fit before the train/test split, a feature built from the target, shuffled cross-validation on a time series, accuracy quoted on data that is 99% one class, a threshold tuned on the test set. The model returns one verdict, OK or PROBLEM, then says why. The set is balanced 15/13, so a model that just yells PROBLEM at everything lands near 50% and gets caught. Then I ran it against a spread of current models on Kaggle. The first thing it caught was not a model. It was the harness. Kaggle runs the models for you on its Model Proxy. When I pulled the raw results, many of the answers were empty. The run finished, but no response was captured. Grading an empty answer as wrong made strong models look broken. gpt-6-astra showed 14%. claude-opus-5 showed 4%. DeepSeek-R1 showed 17%. None of it was real. I re-ran the same prompts, with the same grader, through a second gateway that actually returns the text. DeepSeek-R1 went from 17% to 100%. Same model, same questions, same scoring. The only thing that changed was a harness that handed back the answer. That is the uncomfortable lesson for a benchmarking contest: a benchmark that does not audit its own capture is measuring the plumbing, not the model. So the scorer now flags every empty response and drops it, instead of counting it as wrong. With capture fixed, the picture is more interesting than "big good, small bad." They catch bad code, then cry wolf on the good code. The strong models are near-perfect at flagging a genuine flaw. Where they drop points is the opposite error: calling sound code a PROBLEM. Claude Sonnet 5 and Gemini 3.8 Flash each miss one clean item, both by over-flagging. I expected models to be too trusting. Today's frontier models are the other thing, a nervous reviewer that cannot leave a correct snippet alone. Scale flips the bias. The small model inverts it. Gemma 4 31B keeps 85% on the clean code but its recall on the hidden flaws falls to 20%. It is the one that actually trusts the prose: it reads the confident claim, agrees, then walks past the leak. So the failure mode is not fixed. Big models over-flag, small models under-flag, for opposite reasons. Reasoning is worth about a third of the score and you can see where. Grok 4.20 scores 100% with reasoning on and 68% with it off. The 32 points do not come off evenly. They come off the subtle pairs built to separate understanding from pattern-matching: a parameter-free transform before the split is fine, a fitted scaler before the split is a leak; filling a missing value with a constant is fine, filling it with the column mean is a leak. Turn reasoning off and the model stops telling "before the split" apart from "using numbers from the split." It matches the position of a line instead of what the line computes. | Model | Accuracy | Catches flaws | Clears clean code | |---|---|---|---| | Gemini 3.1 Pro | 100% | 100% | 100% | | Gemini 3.7 Flash | 100% | 100% | 100% | | Grok 4.20 reasoning | 100% | 100% | 100% | | DeepSeek-R1 | 100% | 100% | 100% | | GLM-5 | 100% | 100% | 100% | | Claude Sonnet 5 | 96% | 100% | 92% | | Gemini 3.8 Flash | 96% | 100% | 92% | | Claude Haiku 4.5 | 86% | 100% | 69% | | Grok 4.20 no reasoning | 68% | 100% | 31% | | Gemma 4 31B | 50% | 20% | 85% | The benchmark is public on Kaggle: https://www.kaggle.com/benchmarks/tasks/zkasuran/blog-vs-bytecode/1 https://www.kaggle.com/benchmarks/tasks/zkasuran/blog-vs-bytecode/1 . Every item, its verdict and the grader are open. Kaggle captures each model's answer verbatim, so you can read exactly where it went wrong. Point any model on the Kaggle model list at it. Two backends were used: Kaggle's Model Proxy for the models it captured cleanly and an OpenAI-compatible gateway for the ones the proxy dropped, with the identical prompt and grader. GLM-5 was measured in its FP8 build. GLM-5 and Gemini 3.5 Flash Lite are scored on the items the proxy captured. Grades are on the verdict. The written reason is captured too. On the top models it names the right failure almost every time. AI assistance Claude was used to build the benchmark and draft this post. The design, the items, the grading and the analysis were reviewed and verified by the author, including reading the raw model outputs to catch the capture bug above.