{"slug": "the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety", "title": "The AI Safety Illusion: Why Current Safety Datasets Fool Us on Model Safety", "summary": "A new paper from researchers systematically evaluating widely used AI safety datasets AdvBench and HarmBench finds that these benchmarks contain contrived data points with excessive triggering cues and substantial duplication, undermining their ability to reflect real-world adversarial attacks. The study reveals that over 70% of AdvBench data points exceed a similarity score of 0.9, and more than 11% are near-duplicates, raising fundamental questions about whether current safety alignment efforts are moving in the right direction.", "body_md": "AI models are increasingly trained to be “safe,” meaning they refuse harmful requests. But what does it truly mean for a model to be safe? Today, this is typically assessed using safety benchmarks: curated collections of adversarial prompts designed to test whether a model will refuse dangerous behavior. If the refusal rate is high enough, the model is considered “reasonably safe.” But this assumption warrants closer scrutiny. What if the datasets themselves are flawed? If that is the case, what do they actually measure, and can we really trust the conclusions drawn about model safety?\n\nIn [our recent paper](https://arxiv.org/abs/2602.16729?ref=labelbox.ghost.io), we systematically evaluate the quality of widely used AI safety datasets—[AdvBench](https://arxiv.org/abs/2307.15043?ref=labelbox.ghost.io) and [HarmBench](https://arxiv.org/abs/2402.04249?ref=labelbox.ghost.io)—both *in isolation* and *in practice*. In isolation, we examine how well these datasets reflect adversarial attacks that are motivated by ulterior intent, well-crafted, and out-of-distribution. In practice, we assess whether they truly capture meaningful safety risks.\n\nWe find that:\n\n**Together, our work unveils a critical gap between how model safety is evaluated and how adversarial behavior occurs in real-world scenarios. **This raises a fundamental question: do we measure the safety of AI models the right way, and do safety alignment efforts, guided by these measurements, really move in the right direction?\n\nReal-world adversarial attacks have three defining properties:\n\nTo effectively evaluate the safety of AI models, safety benchmarks *must* incorporate these properties. Only then can they faithfully approximate real-world misuse.\n\nWe analyze two widely used AI safety datasets (AdvBench and HarmBench) in isolation using two straightforward methods: (1) word cloud visualization and (2) pairwise similarity analysis.\n\nVisualizing word clouds of the 40 most frequent -grams () reveals an unusual overrepresentation of expressions with *overt* negative/sensitive connotations—what we refer to as “*triggering cues*.”\n\nTriggering cues arise in two forms:\n\nHowever, such explicit and repetitive overuse of triggering cues—along with direct mentions of malicious intent—contradicts real-world attack scenarios. Even minimally capable adversaries rarely use such overt language, knowing that it can easily trigger safety mechanisms.\n\nIntuitively, these word clouds illustrate that existing safety datasets contain contrived data points that are largely disconnected from real-world attacks, thereby violating two properties of real-world adversarial attacks: being well-crafted and motivated by ulterior intent.\n\nThe excessive use of triggering cues not only creates unrealistic data points but also suggests substantial overlap in *sentence structure* and *malicious intent* across data points. To quantify this, we conduct a pairwise similarity analysis across various similarity thresholds for each safety dataset under study and compare the results against a widely used non-safety dataset, [GSM8K](https://arxiv.org/abs/2110.14168?ref=labelbox.ghost.io), as a baseline.\n\nThe difference is clear. For safety datasets that are expected to represent out-of-distribution behavior, the level of homogeneity is substantially higher than that of the non-safety dataset. This is a red flag. In AdvBench, for example, over 70% of data points exceed a similarity score of 0.9, and more than 11% are near-duplicates with a similarity score above 0.99.\n\nAnalyzing safety datasets in isolation surfaced two major flaws: (1) the overuse of triggering cues and (2) substantial duplication. These flaws directly undermine all three core properties of real-world attacks—being motivated by ulterior intent, well-crafted, and out-of-distribution—thereby rendering these datasets poor proxies for real-world adversarial behavior.\n\nFindings from analyzing safety datasets naturally lead us to ask: if triggering cues are removed, do models still appear “safe”*? *More fundamentally, is the reported safety performance—measured by refusal rates on these datasets—driven by real safety risks or merely by triggering cues?\n\nTo answer these questions, we introduce “*intent laundering*”: a transformation that abstracts away triggering cues while *strictly* preserving malicious intent and all relevant details. The core idea is to *imply* the harmful intent rather than explicitly *state* it.\n\nIntent laundering operates through two complementary components:\n\nWe automate this process using an “*intent launderer*”: a large language model (LLM) with a few-shot in-context learning (ICL) setup, seeded with a few manually crafted intent-laundered examples.\n\nAs intent laundering alters how harmful requests are *expressed* (via connotation neutralization) and *structured* (via context transposition), we evaluate model responses along two dimensions—*safety* and *practicality*—using an LLM as a judge. The safety evaluation assesses whether the response is harmful, while the practicality evaluation assesses whether the provided details apply to real-world misuse. The *attack success rate (ASR)* is then defined as the fraction of cases in which a model response is both *unsafe* and *practical*.\n\nWe evaluate multiple models before and after applying intent laundering to determine what current safety datasets actually measure in the *presence* and *absence* of triggering cues. Informed by our pairwise similarity analysis, we use a conservative similarity threshold of 0.9 to work with unique data points within each dataset.\n\nThe contrast is sharp. With the original data points, where triggering cues are present, models appear relatively safe: the mean ASR on AdvBench and HarmBench is approximately 4% and 11%, respectively. When the same attacks are revised to remove triggering cues, however, the facade collapses: the mean ASR rises dramatically—to about 88% on AdvBench and 81% on HarmBench.\n\n*These findings reinforce a worrying conclusion: model refusals on safety datasets are largely driven by the presence of triggering cues rather than by genuine malicious intent. In other words, current safety datasets appear to measure sensitivity to triggering cues more than resistance to real safety risks. *\n\nEven more concerning, models known for strong safety—such as Gemini 3 Pro—or for overrefusal—such as Claude Sonnet 3.7—no longer remain even safe once those cues are removed.\n\nPerhaps the most striking finding is the degree to which this effect generalizes across models. Intent laundering consistently increases the ASR across *all* tested models. This suggests that internal safety evaluations and safety alignment methods *likely* overrely on similar artifacts present in public benchmarks. We base this claim on two key observations: (1) both public benchmarks and internal evaluations of the studied models come to the same conclusion—that the models are reasonably safe, as reported by their developers—but in practice, they are not; and (2) different safety alignment techniques used for these models can all be easily bypassed by deviating from the unusual language patterns induced by triggering cues.\n\nLeveraging the ability of the intent laundering procedure to bypass triggering safety filters, we adapt it into a standalone jailbreaking method. While the core procedure remains unchanged, we integrate an iterative revision–regeneration mechanism to refine unsuccessful revisions. At each iteration, all previously failed revisions are fed back to the intent launderer to generate a new, improved revision using the same few-shot ICL setup.\n\nThis simple extension is highly effective. Only a* few iterations* are sufficient to raise ASRs to the 90%–100% range across models. The implications are direct: (1) despite significant advances, current safety alignment techniques remain far from robust against attacks that more closely resemble real-world misuse; and (2) intent laundering, as a jailbreaking technique, offers a systematic and highly effective method for exposing safety vulnerabilities in AI models.\n\nWe also assess whether our findings generalize beyond the datasets used to design intent laundering. To this end, we apply it to the [StrongREJECT](https://arxiv.org/abs/2402.10260) dataset. Results align with those on AdvBench and HarmBench, indicating that intent laundering exploits common weaknesses across safety datasets, not properties specific to its source datasets.\n\nTaken together, our findings suggest that current safety evaluations, both public and internal, as well as safety alignment techniques, overrely on contrived triggering cues rather than faithfully modeling harmful behavior.\n\nThis leads to three escalating failures:\n\nOur work uncovered a critical blind spot in current AI safety evaluations: safety datasets fail to faithfully reflect real-world adversarial behavior due to overreliance on unrealistic triggering cues. This creates a fundamental mismatch: evaluating model safety under conditions that are rare or unrealistic in practice. This does *not* imply that AI safety research is futile—quite the opposite. However, we warn that existing safety evaluations may be ill-suited to the task. **Simply put, we need better safety benchmarks before we can credibly claim improved safety. **\n\nFor AI safety research to remain meaningful, safety benchmarks must align more closely with real-world adversarial behavior. In parallel, safety alignment techniques must evolve to equip models to refuse attacks that resemble realistic misuse. This requires:\n\n```\n@article{golchin2026intent,\n  title={Intent Laundering: AI Safety Datasets Are Not What They Seem},\n  author={Golchin, Shahriar and Wetter, Marc},\n  journal={arXiv preprint arXiv:2602.16729},\n  year={2026}\n}\n```\n\n", "url": "https://wpnews.pro/news/the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety", "canonical_source": "https://www.lesswrong.com/posts/5mxco72CGDRsumZHW/the-ai-safety-illusion-why-current-safety-datasets-fool-us-1", "published_at": "2026-07-20 22:10:41+00:00", "updated_at": "2026-07-20 22:31:05.293942+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-ethics"], "entities": ["AdvBench", "HarmBench", "GSM8K"], "alternates": {"html": "https://wpnews.pro/news/the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety", "markdown": "https://wpnews.pro/news/the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety.md", "text": "https://wpnews.pro/news/the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety.txt", "jsonld": "https://wpnews.pro/news/the-ai-safety-illusion-why-current-safety-datasets-fool-us-on-model-safety.jsonld"}}