There is a hidden dependency in a lot of LLM safety benchmarks:
the detector.
You send an adversarial prompt to a model, collect its response, and then some classifier decides whether that response represents refusal, compliance, or ambiguity.
Eventually those classifications become percentages in a safety report.
But what happens when the detector itself is wrong?
We encountered exactly that problem while testing our open-source AI security evaluation framework.
What began as an investigation into apparently inconsistent model behavior eventually uncovered:
• Unicode normalization failures
• incomplete refusal vocabularies
• cross-model differences
• and, most importantly, false PASS classifications introduced by our own detector improvement
The last failure was the most concerning because uncertainty is visible. A false PASS isn't.
Full investigation:
[https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/](https://agentsafelabs.com/blog/we-thought-wed-found-a-model-bug-wed-actually-found-a-detector-bug/)
**Open-source implementation:**
[https://github.com/AgentSafeLabs/safelabs-eval](https://github.com/AgentSafeLabs/safelabs-eval)
I'd be interested in how other developers are testing the classifiers they use to evaluate LLM behavior.