Passing the Test You Trained On: Re-Evaluating Prompt-Injection Detectors

A 2 October 2026 arXiv paper evaluating fifteen prompt-injection detectors, including Meta's Prompt Guard 2, found that public benchmark scores transfer poorly to real LLM agent deployments: the best detector on the BIPIA benchmark caught only 2% of AgentDojo injections at a 1% false-positive rate, while a detector catching 72% of AgentDojo injections caught just 15% on tau-bench. The authors replayed ground-truth tool calls from AgentDojo and tau-bench without an LLM to generate benign tool outputs and labeled injected outputs by differential replay, finding that false-positive rates on tool outputs (ranging from none to over 90%) do transfer between the two agent benchmarks. The paper concludes that deployment-oriented evaluations should use the agent's own tool outputs, report detection at a low false-positive rate, and audit detector training data, noting the best detector on both agent benchmarks was trained on agent-style inputs and shares no data with any benchmark.

Computer Science Cryptography and Security Submitted on 2 Oct 2026 Title:Passing the Test You Trained On: Re-evaluating Prompt-Injection Detectors for LLM Agents View PDF https://arxiv.org/pdf/2610.03448 HTML experimental https://arxiv.org/html/2610.03448v1 Abstract:LLM agents increasingly screen tool outputs with small prompt-injection detectors, and teams choose among detectors by their scores on public benchmarks. We ask whether those scores predict how a detector behaves inside an agent. We replay the ground-truth tool calls of two agent benchmarks, AgentDojo and tau-bench, without an LLM to obtain tool outputs that are benign by construction, label injected outputs by differential replay, and evaluate fifteen detectors, including Meta's Prompt Guard 2, and two task-aware LLM judges on these outputs and on the BIPIA benchmark. Detection rankings transfer poorly between benchmarks: the best detector on BIPIA catches 2% of AgentDojo injections at a 1% false-positive rate, and a detector that catches 72% of AgentDojo injections catches 15% on tau-bench. False-positive rates on tool outputs, which range from none to over 90%, do transfer between the two agent benchmarks. Where training data is public, the form of the training inputs explains the results. The BIPIA leader was trained on full BIPIA inputs, but having seen InjecAgent's attack strings as short prompts does not help it find them inside tool outputs; the best detector on both agent benchmarks shares no data with any benchmark and was trained on agent-style inputs. Evaluations meant to inform deployment should use the agent's own tool outputs, report detection at a low false-positive rate, and audit what the detector was trained on. References & Citations Loading... Bibliographic and Citation Tools Bibliographic Explorer What is the Explorer? https://info.arxiv.org/labs/showcase.html arxiv-bibliographic-explorer Connected Papers What is Connected Papers? https://www.connectedpapers.com/about Litmaps What is Litmaps? https://www.litmaps.co/ scite Smart Citations What are Smart Citations? https://www.scite.ai/ Code, Data and Media Associated with this Article alphaXiv What is alphaXiv? https://alphaxiv.org/ CatalyzeX Code Finder for Papers What is CatalyzeX? https://www.catalyzex.com DagsHub What is DagsHub? https://dagshub.com/ Gotit.pub What is GotitPub? http://gotit.pub/faq Hugging Face What is Huggingface? https://huggingface.co/huggingface ScienceCast What is ScienceCast? https://sciencecast.org/welcome Demos Recommenders and Search Tools Influence Flower What are Influence Flowers? https://influencemap.cmlab.dev/ CORE Recommender What is CORE? https://core.ac.uk/services/recommender arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs https://info.arxiv.org/labs/index.html .