{"slug": "llm-safety-why-asr-is-a-bad-metric-for-defenders", "title": "LLM Safety: Why ASR is a Bad Metric for Defenders", "summary": "A new framework called A-MESS, which uses AttackSHAP Shapley-based scoring, shows that Attack Success Rate (ASR) is a poor metric for improving LLM safety because high ASR correlates weakly with actual safety utility. The findings demonstrate that selecting a compact subset of high-utility attacks is significantly more effective than including all successful attacks in training, making the approach practical for real-world LLM agent development.", "body_md": "# LLM Safety: Why ASR is a Bad Metric for Defenders\n\n*every*successful attack; you need the ones that actually move the needle on model resilience.\n\nThis is where a defender-centric approach changes the game. Instead of asking \"Did this break the model?\", the question becomes \"Does including this attack in my training set actually improve downstream safety?\"\n\nThe A-MESS framework tackles this by using AttackSHAP—a Shapley-based scoring system. For those unfamiliar, Shapley values are typically used in game theory to determine how much each player contributes to a total outcome. In this context, it treats individual jailbreak attacks as \"players\" and calculates their marginal utility toward improving the model's safety.\n\nThe findings here are pretty revealing:\n\n**ASR vs. Utility:** High attack success rates correlate weakly with actual safety utility. A \"noisy\" jailbreak might have a high ASR but provide zero value for alignment training.**Subset Optimization:** Selecting a compact subset of high-utility attacks is significantly more effective than just dumping every successful attack into the training pipeline.**Efficiency:** AttackSHAP can be estimated accurately without needing an infinite number of utility queries, making it practical for real-world LLM agent development.\n\nEssentially, we need to stop treating jailbreaks as just \"failures\" to be patched and start treating them as high-value data resources. The goal isn't just to stop one specific prompt, but to identify the underlying vulnerability that the prompt exposed.\n\nIf you're building a safety pipeline from scratch, focusing on marginal utility rather than raw success rates will likely lead to a much leaner and more effective red-teaming dataset.\n\n[Next Preemptive Hardening for Agentic LLM Security →](/en/threads/2326/)", "url": "https://wpnews.pro/news/llm-safety-why-asr-is-a-bad-metric-for-defenders", "canonical_source": "https://promptcube3.com/en/threads/2345/", "published_at": "2026-07-23 13:50:29+00:00", "updated_at": "2026-07-23 22:05:50.150760+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-research"], "entities": ["A-MESS", "AttackSHAP"], "alternates": {"html": "https://wpnews.pro/news/llm-safety-why-asr-is-a-bad-metric-for-defenders", "markdown": "https://wpnews.pro/news/llm-safety-why-asr-is-a-bad-metric-for-defenders.md", "text": "https://wpnews.pro/news/llm-safety-why-asr-is-a-bad-metric-for-defenders.txt", "jsonld": "https://wpnews.pro/news/llm-safety-why-asr-is-a-bad-metric-for-defenders.jsonld"}}