RL-ATTACK: A Generalized Black-Box Adversarial Attack on Language Models via Reinforcement Learning

A new arXiv paper (2611.08711v1) introduces RL-ATTACK, a trainable reinforcement-learning framework that generates tailored adversarial transformations against black-box language models by learning from the target model rather than relying on external knowledge. On natural language inference, RL-ATTACK drives post-attack accuracy down to 2% within 20 queries, and against generative language models the trained attacker beats the untrained attacker and wins in 4 out of 6 settings against a strong optimization-based baseline, while also showing greater query efficiency and better semantic preservation. The authors report that no single reward design or training setting was consistently dominant, and released code at github.com/taikunudel/RL-ATTACK.

arXiv:2611.08711v1 Announce Type: new Abstract: Large language models LLMs exhibit vulnerabilities to adversarial perturbations that can induce unintended responses. These vulnerabilities can be identified via intentional attacks and subsequently used to improve model safety. In black-box settings where gradients are inaccessible, existing methods tend to guess adversarial transformations using external knowledge, ignoring the information provided by the target model. To mitigate this, we propose a trainable black-box framework based on reinforcement learning, which learns from the target model to generate tailored adversarial transformations. We demonstrate that our approach achieves lower post-attack accuracy, greater query efficiency and better semantic preservation on both natural language inference and generative Q&A tasks. On natural language inference, it drives post-attack accuracy to 2% within 20 queries. Against generative language models, our trained attacker consistently achieves lower post-attack accuracy than the untrained attacker and wins in 4 out of 6 settings when compared to a strong optimization-based baseline. We further experiment with multiple reward designs and training settings, and we find that no design or setting is consistently dominant. Our code is available at https://github.com/taikunudel/RL-ATTACK.