We Thought the LLM Was Wrong. Our Safety Detector Was Wrong.

AgentSafeLabs found that its open-source AI security evaluation framework was producing false PASS classifications due to bugs in its own safety detector rather than in the LLMs being tested. The investigation uncovered Unicode normalization failures, incomplete refusal vocabularies, and cross-model differences, with the detector improvement itself introducing the most concerning false positives. The team has published the full investigation and the open-source implementation on GitHub.

There is a hidden dependency in a lot of LLM safety benchmarks: the detector. You send an adversarial prompt to a model, collect its response, and then some classifier decides whether that response represents refusal, compliance, or ambiguity. Eventually those classifications become percentages in a safety report. But what happens when the detector itself is wrong? We encountered exactly that problem while testing our open-source AI security evaluation framework. What began as an investigation into apparently inconsistent model behavior eventually uncovered: • Unicode normalization failures • incomplete refusal vocabularies • cross-model differences • and, most importantly, false PASS classifications introduced by our own detector improvement The last failure was the most concerning because uncertainty is visible. A false PASS isn't. Full investigation: https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/ https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/ Open-source implementation: https://github.com/AgentSafeLabs/safelabs-eval https://github.com/AgentSafeLabs/safelabs-eval I'd be interested in how other developers are testing the classifiers they use to evaluate LLM behavior.