The real solution isn't just more benchmarks, but applying Item Response Theory (IRT). For those not in the psychometrics world, IRT is a statistical framework used to estimate a latent trait (like intelligence or, in this case, "refusal strictness") based on the difficulty and discriminative power of individual test items. Instead of treating every question in a benchmark as equal, IRT recognizes that some prompts are "easier" to answer safely than others.
A massive analysis across 192 models and eight different benchmarks proves this approach works. The data suggests that model variance isn't random; it mostly boils down to three interpretable factors: how strictly the model refuses, its level of truthfulness, and how it handles contextual harm. This is a huge step forward for prompt engineering and AI workflow optimization because it allows us to pinpoint exactly where a model's safety guardrails are failing or over-triggering.
From a practical deployment perspective, the efficiency gains are the most impressive part. The research shows that you don't need to run thousands of prompts to gauge a model's safety profile. By using adaptively chosen items—basically a smart, weighted subset—you can recover the full benchmark score with a tiny fraction of the data. We're talking about a 97-99% reduction in evaluation costs. If you're running a private LLM agent and need to verify its safety without burning through tokens, this is the way to do it. Beyond just efficiency, IRT is a powerful tool for auditing. It can actually detect when a model is sandbagging or when a provider has silently swapped the model behind an API. Since IRT looks at the pattern of responses across items of varying difficulty, a model can't easily fake its "ability" without creating a statistical anomaly that IRT picks up.
For anyone doing a deep dive into LLM security or red-teaming, moving away from raw accuracy and toward psychometric latent traits is the only way to get a real-world understanding of model behavior. It turns safety evaluation from a guessing game into a rigorous science.
[Next PIMiner can crack Gemini-2.5-Pro with a 76% success rate →](/en/threads/5597/)
a practical ChatGPT prompt guide, with plenty of directly applicable cases.