The Fair-ASR paper finally addresses this. Their protocol fixes a shared target-call budget B across all attacks, then tracks attacker-side calls separately for efficiency. Target calls are the only directly observable, method-agnostic resource when you're hitting a closed model. FLOPs estimates are fantasy for black-box systems.
They re-evaluated 11 representative attacks under this regime. The rankings shuffle violently depending on B:
Low budget (B=5-10): Hand-crafted templates and simple stochastic perturbations (PAIR, AutoDAN) dominate. LLM-driven agents waste their budget on planning overhead.Medium budget (B=20-50): Gradient-free optimization methods start catching up, but the gap narrows.High budget (B=100+): The compute-heavy agents finally justify their cost, but by then you've spent hundreds of dollars per target.
No evaluated method was efficient in both dimensions. Either you burn target calls or you burn attacker calls (local LLM invocations). That tradeoff is the real story.
Enter ReCode. The authors built it by composing two cheap primitives Fair-ASR surfaced: desensitization rewriting + a lightweight template mutation. At B=20 target calls against GPT-5:
85% ASR****7.19 attacker calls per request average
That's the first method in their suite sitting in the Pareto frontier for both metrics. The rewriting step strips safety triggers without changing semantic intent, then the template layer probes the weakened guardrails. Total local compute stays trivial — you could run the attacker side on a consumer GPU.
What this means for red-teaming: stop reporting raw ASR. Publish your B. Publish attacker-call counts. A "95% ASR" that requires 200 target queries and 500 local LLM calls is a science project, not a vulnerability. Fair-ASR gives us the vocabulary to compare apples to apples.
The paper also hints at a deeper issue: current safety training optimizes for single-turn robustness. Multi-turn, budget-constrained compositional attacks like ReCode exploit the gap between per-turn safety and cumulative exposure. That's where the next evaluation cycle needs to go.
Next Tripwire actually manages to kill jailbreaks without →