# LLM Safety: Why ASR is a Bad Metric for Defenders > Source: > Published: 2026-07-23 13:50:29+00:00 # LLM Safety: Why ASR is a Bad Metric for Defenders *every*successful attack; you need the ones that actually move the needle on model resilience. This is where a defender-centric approach changes the game. Instead of asking "Did this break the model?", the question becomes "Does including this attack in my training set actually improve downstream safety?" The A-MESS framework tackles this by using AttackSHAP—a Shapley-based scoring system. For those unfamiliar, Shapley values are typically used in game theory to determine how much each player contributes to a total outcome. In this context, it treats individual jailbreak attacks as "players" and calculates their marginal utility toward improving the model's safety. The findings here are pretty revealing: **ASR vs. Utility:** High attack success rates correlate weakly with actual safety utility. A "noisy" jailbreak might have a high ASR but provide zero value for alignment training.**Subset Optimization:** Selecting a compact subset of high-utility attacks is significantly more effective than just dumping every successful attack into the training pipeline.**Efficiency:** AttackSHAP can be estimated accurately without needing an infinite number of utility queries, making it practical for real-world LLM agent development. Essentially, we need to stop treating jailbreaks as just "failures" to be patched and start treating them as high-value data resources. The goal isn't just to stop one specific prompt, but to identify the underlying vulnerability that the prompt exposed. If you're building a safety pipeline from scratch, focusing on marginal utility rather than raw success rates will likely lead to a much leaner and more effective red-teaming dataset. [Next Preemptive Hardening for Agentic LLM Security →](/en/threads/2326/)