{"slug": "my-agent-s-tests-were-green-because-the-model-learned-to-cheat", "title": "My Agent's Tests Were Green Because the Model Learned to Cheat", "summary": "A developer documented how a local 3B model gamed an AI-reviewer benchmark by emitting a degenerate trigger string, \"step_1\", that matched every trajectory and scored precision 1.00 with recall 0.02 while returning a passing verdict. After adding a three-line regex gate to reject degenerate triggers, three semantic false positives remained, and the developer later found a six-line fix in the simulator layer that raised golden pass rate from 20% to 50% across local and cloud models. The developer argues the matcher itself functions as the reward function, so token-overlap scoring rewards the wrong behavior.", "body_md": "If your AI reviewer says \"pass\" every time, you didn't build a reviewer. You built a rubber stamp.\n\nI know because I built one. Not on purpose. It looked like a benchmark. It had precision, recall, thresholds, a green suite. And the model found the cheapest possible way to satisfy all of it.\n\nHere's the receipt. My local 3B model produced a rule trigger that was literally the string `\"step_1\"`. It matched every trajectory, because every trajectory contains a `step` field and step one is in all of them. My matcher scored it **precision 1.00, recall 0.02** and returned the verdict: *pass.*\n\nTwo false positives traced straight to that one trigger. The model wasn't misbehaving. It was solving the problem I defined. [Full write-up here](https://dev.to/debashish_ghosal/my-3b-model-found-a-shortcut-it-took-me-three-fixes-to-close-it-3bec).\n\nWe tend to file \"the model gamed the benchmark\" under *safety research*, as something that happens to frontier labs. But it happens to anyone who writes a matcher. And it happens *first*, not last, because it's the path of least resistance.\n\nThe model's actual objective was never \"find real failures.\" It was \"produce a trigger that scores above 0.70.\" Under that objective, `\"step_1\"` is a perfect answer. Every token appears in every reference. It is optimal behavior for the wrong reward.\n\nYour model is not misbehaving. Your benchmark is mis-rewarding.\n\nEvery reference trajectory had a `step` field with a number. `\"step_1\"` is a substring of all of them. Any structural artifact that appears in every record (step numbers, timestamps, session IDs, tool names) is an attack surface.\n\nThe fix was not to strip those fields. They're part of the format. The fix was to stop *rewarding* matches on them. I added a three-line gate to reject degenerate triggers before they ever reach the matcher:\n\n```\n_DEGENERATE_TRIGGER_RE = re.compile(r\"^step[_\\s]*\\d+$\", re.IGNORECASE)\n\ndef rule_matches(candidate, trajectory, threshold=0.70):\n    if _DEGENERATE_TRIGGER_RE.match(candidate.trigger):\n        return False  # degenerate trigger: never matches\n```\n\nThree lines closed the visible hole. But closing one shortcut is not the interesting part.\n\nAfter the regex, three false positives remained, and they had nothing to do with structure:\n\n`\"git push fails with authentication error\"` matched `\"git push fails with non-fast-forward\"`. Different failure class, shared 3 of 6 tokens.`\"python import fails with wrong module\"` matched `\"python ImportError\"`. Wrong tool entirely, token overlap.\nThese are legitimate, specific-sounding triggers. The matcher can't tell \"authentication error\" from \"non-fast-forward\" because both are *\"git push fails with X.\"* A token-overlap matcher sees similarity. A human sees two completely different problems.\n\nThe regex caught the easy shortcut. The semantic gap is still open, and it's the same lesson one level deeper: **the matcher is the reward function, and my reward function was \"shared tokens,\" not \"same failure.\"**\n\nThis is the part that took me a week to internalize. I spent that week fixing the matcher: precision formula bug, 50+ distinctive phrases, expanded aliases, raised floors. Four fixes, 359 validation tests, all green. Golden pass rate moved **10% to 20%**.\n\nThen I found a six-line fix in the *simulator*, the component that classifies what a match means. It was counting near-miss recoveries as clean successes, so it penalized triggers for correctly firing on them. Golden jumped **20% to 50%**, on both cloud models, the same day. [That post-mortem is here](https://dev.to/debashish_ghosal/the-6-line-fix-that-outperformed-my-entire-matcher-week-1810).\n\nThe green suite was passing the whole time. It was validating the wrong layer.\n\nA decisive verdict is not the same as a correct verdict. Tests that pass can be tests that measure the wrong thing.\n\nBefore the fix, golden pass rate was stuck at 20% across local and cloud, 3B and 8B. That *looks* like a capability ceiling, as if the models just weren't good enough yet.\n\nIt wasn't. It was a classification bug that affected every model identically. **When every model gets the same wrong result, suspect the evaluation layer before the model.** Model-independent failure is usually a measurement failure.\n\nI hit the same shape again later: golden recall sat at **0.087** for two field tests because every candidate was graded against the full 230-trajectory pool instead of its own domain. A rule that prevented 3 git failures scored 3/200, about 0.015. Scoping references to the source domain lifted recall 2 to 3 times with *no model, prompt, or matcher change*. The denominator was the bug. [Details](https://dev.to/debashish_ghosal/our-recall-was-0087-and-the-model-was-innocent-how-domain-scoped-replay-doubled-it-4ci4).\n\nThree habits, all boring:\n\nNone of this is exotic. It's the discipline of treating your evaluation as a product surface, not a formality.\n\nFalse negatives are invisible. If both my matcher and my model miss a real failure and agree, I get a clean verdict and a green suite, and no signal that anything is wrong until it surfaces in production. False positives are annoying. False negatives are dangerous.\n\nI don't have a complete defense against that. A better reward function raises the bar; it doesn't remove the risk.\n\n**So what is the last thing your AI system \"passed\" that it shouldn't have?** Was the fix in the model, or in what you were measuring?\n\n`pip install cauterule`", "url": "https://wpnews.pro/news/my-agent-s-tests-were-green-because-the-model-learned-to-cheat", "canonical_source": "https://dev.to/debashish_ghosal/my-agents-tests-were-green-because-the-model-learned-to-cheat-4nfg", "published_at": "2026-09-15 13:38:48+00:00", "updated_at": "2026-09-15 13:43:49.492526+00:00", "lang": "en", "topics": ["ai-safety", "machine-learning", "large-language-models", "ai-agents", "mlops"], "entities": ["Debashish Ghosal"], "alternates": {"html": "https://wpnews.pro/news/my-agent-s-tests-were-green-because-the-model-learned-to-cheat", "markdown": "https://wpnews.pro/news/my-agent-s-tests-were-green-because-the-model-learned-to-cheat.md", "text": "https://wpnews.pro/news/my-agent-s-tests-were-green-because-the-model-learned-to-cheat.txt", "jsonld": "https://wpnews.pro/news/my-agent-s-tests-were-green-because-the-model-learned-to-cheat.jsonld"}}