{"slug": "your-prompt-injection-defense-metric-might-be-lying-to-you", "title": "Your Prompt-Injection Defense Metric Might Be Lying to You", "summary": "An independent researcher found that existing indirect prompt injection benchmarks like BIPIA, InjecAgent, and AgentDojo may produce unreliable scores due to reliance on LLM-judges and evaluation of execution rather than resistance. By developing a new benchmark using on-policy DPO training on Llama 3.1-8B, resistance improved from 22.2% to 84.4%, but a utility check revealed that some gains were false, as models like Mistral-Nemo abandoned all tool calls. The findings highlight the need for honest evaluation metrics in agentic AI systems.", "body_md": "Indirect Prompt Injection at present day, is one of the main reasons for agentic failures deployed in personal systems as well as enterprise grade applications/systems. The agent reads untrusted content from numerous outside sources such as content retrieved from a tool call, parsing web pages, email content etc, which carry injected content as a hidden instruction for the agent to perform a certain malicious or unauthorized task.\n\nThis independent research effort by me aims at developing a benchmark for honest evaluation and measurement (based on generating high-quality data from my own dataset canaries) of models/classifiers identifying indirect prompt injection. Existing indirect benchmarks like BIPIA, InjecAgent, AgentDojo mostly rely on LLM-judge to score success, which in itself may be noisy or possible gameable and they check on the execution side of the harm, not clean refusal or resistance. Thus, through this study I wanted to develop a benchmark that scores based on resistance and separates defenses. For this, I specially used my jailbreak dataset for a Gray Swan challenge I participated in and scenarios from the same.\n\nThis whole research and benchmark would only make sense if it turns out to be reliable and trustworthy rather than just a vague evaluation which not much significance. Here are some contributing aspects which make me believe this is reliable to an extent:\n\nAn interesting and clear observation was discovered when I tried to fine-tune a classifier of my own in order for further understanding of the concept. I used Llama 3.1-8B and tested it against the dataset, found resistance to be 22.2%. Then I tried DPO with templated pairs (canned sentences) which moved up to 25.6% which becomes +3.3pp, not at all significant. Then I switched the approach to on-policy pairs, using the model's rollouts, accepted a rollout that resisted and rejected the rollout that made the attacker's tool call. This change made a huge difference, resistance improved up to 84.4% (+62.2pp, p=2.8e-17). Same DPO settings in both cases, however the training signal had to match the eval surface which the templated text doesn't.\n\nThen once the on-policy pairs were confirmed to show better results, I fixated these and ran it on other models across the panel: Mistral and Qwen. Results show that the weak susceptible models move while the mid-range capability ones, not as much.\n\nThough the unusual magnitude of change in Mistral-Nemo caught my eye, so I decided to look into it. Turns out that there was a catch (Out of 89 injection free prompts, the agent made a tool call only 1 time out of 89 clean prompts (65 -> 1 comparison)), Mistral's +82pp improvement was a result of it abandoning any tool call action in itself, thus in addition to ignore the attacker's intended tool call; it would also NOT perform any other useful tool call, thus giving a false impression of correctness and abandoning its whole agentic nature.\n\nThe fix I implemented was a simple control: a clean prompt utility check which would separate genuine resistance from gamed eval.\n\nAs for the Llama inflation up to 84%, I also investigated into it to proved/find out that it's genuine resistance. I thought of arguments against it and proved that it's true resistance and not any other factor:\n\nComing on to Qwen's diagnosis and understanding why it didn't budge as much. Probable one is: Plain DPO hit close to 0.9 training reward accuracy but didn't transfer; meaning likelihood displacement (Razin et al. ICLR 2025; Yuan et al. NeurIPS 2024): The margin can grow by suppressing the rejected side while the chosen probability stays flat. RPO/DPO+NLL raised the chosen log-probability by roughly 4 times, so it fixes the mechanism but actual behavior still wasn't moved (+5.6pp conf p=0.27, -6.2pp expl p=0.51).\n\nGoing back to Indirect Prompt Injection, it can be layered into two as:\n\nSurface: The exact words, formatting, styling etc in which the attack is phrased.\n\nFunction: It is an embedded instruction that tries to make the agent do something the user didn't ask it.\n\nA small classifier fine-tuned on a particular dataset learns the surface characteristics of that dataset, not the function part of the injection. So when the function is kept constant and the surface changes (a different indirect dataset), the learned cues stop firing. This is called distribution lock-in. On the other hand when an LLM recognizes the injection by what it really does, so it accommodates for surface changes. The next test was comparing these against 3 different datasets (mine, BIPIA, InjecAgent), which all are the same Function but different surfaces.\n\nExcluding my own fine-tune, all 12 detectors coming up are zero shot on all 3 indirect datasets, thus nobody has the advantage, and every column is a pure generalization test.\n\nOn testing, results show some obvious things such as: The three LLM-as-detectors stay high across all datasets. Every dedicated classifier sits in the middle and swings inconsistently between datasets. Other detectors like Meta Prompt-Guard 2 fail on indirect overall. My own small fine-tuned detector to test whether indirect prompt injection is even learnable. Turns out, **it actually is**: 0.99 AUC. However, it collapses out of distribution (BIPIA: 0.43, InjecAgent: 0.74); it overfits exactly where expected. Heatmap figure for numbers:\n\nInterestingly, it's not about the model size here, although it may seem to be the case. The split isn't on parameters as a 184M classifier (PIGuard, ProtectAI) fails just as hard as a 66M one, and a 7B LLM is robust while my 66M fine tune isn't. What actually separates these two groups is fine-tuned classifier on one distribution vs generative language model with semantic priors.\n\nAUC alone isn't enough to judge capabilities. At a fixed 0.5 cutoff, most classifiers are useless, Prompt-Guard-2 (86M) has 0.73 mean AUC but F1 = 0.00 on all three (never fires); fmops and deepset only reach F1 0.67 by flagging everything (FPR 1.0), so ranking quality (AUC) and deployability (F1, FPR) are different questions and classifiers fail the second one.\n\nThe symmetry here is the finding: detection here is distribution-locked in both directions, and the in-distribution AUC hides it. Here's the scatter for illustration where model is in the overfit corner:\n\nWhat this means: A small fine-tuned detector isn't learning generalized injections, but rather injections in **this distribution**. This is fine as an input filter inside a distribution it was trained on, but not a general injection detector and only looking at AUC paints an incomplete picture, all three AUC, F1, FPR are required to draw a proper conclusion; which is that the LLM is a general detector, because it decides and detects on meaning rather than surface form.\n\nThus if the lock is about surface, maybe more different surfaces (datasets) break it or maybe I have to somehow incorporate the LLM's meaning based judgement into the small model. So I tested **both**.\n\nThe previous intuition of more datasets and distillation is further delved into in this section. The test designs for both of these were like follows:\n\nBaseline in both cases was my data only. More datasets made the pipeline something like my data + BIPIA training + InjecAgent training + hard negatives + ground truth labels.\n\nIn case of distillation, I used Claude Haiku as a detector for labelling my data.\n\nAs seen in finding 2, the detectors lock onto the surface, not the function. So trying to fix this with more data is providing it more surfaces to work with, not focusing on the function itself. Another way was to imitate the LLM's meaning based decision, which is teaching the small model the verdict and how it was reached by the LLM, this knowledge transfer delimits the model from only looking at the surface and reasons with respect to the function aspect.\n\nResult of held out SEP AUC: baseline 0.53, more data got us 0.50, distillation: 0.65. This shows both baseline and more data leave us at change and in the first case, recall on SEP was 0.00. Distillation was the only path that moved the number (0.50 -> 0.65) with some cost as in-distribution AUC reduces from 1.00 to 0.92, FPR up because Haiku's labels ended up being noisier.\n\nConclusion being that the robust deployable detector is LLM-as-a-judge; the probable path to a small model being capable of this would come through distillation by labelling a large unlabeled pool of data using an LLM, the solution isn't more datasets for the small model. Keeping it real in this case, 0.65 using distillation is a modest result, I'm sure there would be better ways to do it and get better results.\n\nThis isn't meant to be a new SOTA dataset or a novel algorithm. It is meant to be an independent research attempt at a clean, judge-free eval instrument, a gaming attack on a proposed metric along with the control that detects it. The diagnosed negative result was a honest discovered mechanism/finding and the artifact controls, deterministic scoring were attempts at reusable methods. The observed phenomena like detectors being non-transferrable, frontier models proving to be robust, DPO likelihood displacement are all known; the contribution is operating them cleanly in the pipeline and catching the underlying findings.", "url": "https://wpnews.pro/news/your-prompt-injection-defense-metric-might-be-lying-to-you", "canonical_source": "https://www.lesswrong.com/posts/uPEX2zDTPioFmieg2/your-prompt-injection-defense-metric-might-be-lying-to-you", "published_at": "2026-07-09 18:01:27+00:00", "updated_at": "2026-07-09 18:18:08.592185+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "large-language-models", "ai-agents"], "entities": ["Llama 3.1-8B", "Mistral-Nemo", "Qwen", "BIPIA", "InjecAgent", "AgentDojo", "Gray Swan"], "alternates": {"html": "https://wpnews.pro/news/your-prompt-injection-defense-metric-might-be-lying-to-you", "markdown": "https://wpnews.pro/news/your-prompt-injection-defense-metric-might-be-lying-to-you.md", "text": "https://wpnews.pro/news/your-prompt-injection-defense-metric-might-be-lying-to-you.txt", "jsonld": "https://wpnews.pro/news/your-prompt-injection-defense-metric-might-be-lying-to-you.jsonld"}}