{"slug": "fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls", "title": "Fair-ASR flips jailbreak rankings when you equalize target calls", "summary": "A new evaluation protocol called Fair-ASR, introduced in a paper, equalizes target-call budgets across jailbreak attacks and finds that rankings of 11 representative attacks shift dramatically depending on the budget, with no method efficient in both target and attacker calls. The authors propose ReCode, which combines desensitization rewriting and lightweight template mutation, achieving 85% attack success rate (ASR) at 20 target calls against GPT-5 with an average of 7.19 attacker calls per request, placing it on the Pareto frontier for both metrics. The paper urges red-teaming to report target-call budgets and attacker-call counts instead of raw ASR, and highlights that current safety training optimizes for single-turn robustness, leaving gaps for multi-turn compositional attacks.", "body_md": "# Fair-ASR flips jailbreak rankings when you equalize target calls\n\nThe Fair-ASR paper finally addresses this. Their protocol fixes a shared target-call budget B across all attacks, then tracks attacker-side calls separately for efficiency. Target calls are the only directly observable, method-agnostic resource when you're hitting a closed model. FLOPs estimates are fantasy for black-box systems.\n\nThey re-evaluated 11 representative attacks under this regime. The rankings shuffle violently depending on B:\n\n**Low budget (B=5-10):** Hand-crafted templates and simple stochastic perturbations (PAIR, AutoDAN) dominate. LLM-driven agents waste their budget on planning overhead.**Medium budget (B=20-50):** Gradient-free optimization methods start catching up, but the gap narrows.**High budget (B=100+):** The compute-heavy agents finally justify their cost, but by then you've spent hundreds of dollars per target.\n\n**No evaluated method was efficient in both dimensions.** Either you burn target calls or you burn attacker calls (local LLM invocations). That tradeoff is the real story.\n\nEnter ReCode. The authors built it by composing two cheap primitives Fair-ASR surfaced: desensitization rewriting + a lightweight template mutation. At B=20 target calls against GPT-5:\n\n**85% ASR****7.19 attacker calls per request average**\n\nThat's the first method in their suite sitting in the Pareto frontier for both metrics. The rewriting step strips safety triggers without changing semantic intent, then the template layer probes the weakened guardrails. Total local compute stays trivial — you could run the attacker side on a consumer GPU.\n\nWhat this means for red-teaming: stop reporting raw ASR. Publish your B. Publish attacker-call counts. A \"95% ASR\" that requires 200 target queries and 500 local LLM calls is a science project, not a vulnerability. Fair-ASR gives us the vocabulary to compare apples to apples.\n\nThe paper also hints at a deeper issue: current safety training optimizes for single-turn robustness. Multi-turn, budget-constrained compositional attacks like ReCode exploit the gap between per-turn safety and cumulative exposure. That's where the next evaluation cycle needs to go.\n\n[Next Tripwire actually manages to kill jailbreaks without →](/en/threads/6819/)", "url": "https://wpnews.pro/news/fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls", "canonical_source": "https://promptcube3.com/en/threads/6960/", "published_at": "2026-08-19 18:13:09+00:00", "updated_at": "2026-08-19 18:58:37.690204+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models", "ai-agents"], "entities": ["Fair-ASR", "ReCode", "GPT-5", "PAIR", "AutoDAN"], "alternates": {"html": "https://wpnews.pro/news/fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls", "markdown": "https://wpnews.pro/news/fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls.md", "text": "https://wpnews.pro/news/fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls.txt", "jsonld": "https://wpnews.pro/news/fair-asr-flips-jailbreak-rankings-when-you-equalize-target-calls.jsonld"}}