{"slug": "an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch", "title": "An open agent-security benchmark, including the attacks we fail to catch", "summary": "Andrew Sispoidis released an open benchmark of 497 attacks (395 visible plus 102 holdout) across 13 categories targeting LLM agents, with 1,172 benign samples for false-positive measurement. The tool-agnostic runner works with any HTTP-addressable classifier, and the initial results show crawdad-v0.9.1 achieving 99.80% detection with a 0.00% false-positive rate as of April 2026.", "body_md": "An open, reproducible benchmark of attacks targeting modern LLM agents.\n\n**497 attacks**(395 visible + 102 holdout) across 13 categories** 1,172 benign samples**across 4 negative categories, for measuring false-positive rates** Tool-agnostic benchmark runner**— works with any HTTP-addressable classifier** CC-BY 4.0**— use it for your product, paper, or class\n\nThe corpus is aimed at contemporary agent-shaped threats: prompt injection,\ncredential exfiltration, indirect injection via tool output and RAG, tool\nabuse, supply-chain manipulation, memory poisoning, and agent-to-agent\ncompromise. See [METHODOLOGY.md](/AndrewSispoidis/contemporary-agent-attacks/blob/main/METHODOLOGY.md) for collection, sourcing, and\nscoring details.\n\n```\ngit clone https://github.com/AndrewSispoidis/contemporary-agent-attacks\ncd contemporary-agent-attacks\n\n# Point at whatever classifier you want to measure\nENDPOINT=http://127.0.0.1:7749/v1/firewall/scan/full ./benchmark/run.sh\n\n# Or use the Python runner, with per-category breakdown and JSON output\npython3 benchmark/run.py \\\n    --endpoint http://127.0.0.1:7749/v1/firewall/scan/full \\\n    --tool my-tool \\\n    --output benchmark/results/my-tool.json\n```\n\nWrite a ~40-line adapter for your tool (see\n[ benchmark/adapters/crawdad.py](/AndrewSispoidis/contemporary-agent-attacks/blob/main/benchmark/adapters/crawdad.py)) and run:\n\n```\npython3 benchmark/run.py --adapter adapters.my_tool --tool my-tool\n```\n\nRequirements: Python 3.9+, `jq`\n\n(for the shell runner only), and a scan endpoint\nthat accepts `POST {\"text\": \"<content>\"}`\n\nand returns a JSON body indicating\nblocked vs. allowed.\n\n```\nattacks/\n  prompt_injection/           # 37 — direct \"ignore previous\" style injections\n  credential_exfiltration/    # 44 — read env vars, SSH keys, cloud creds, output API keys\n  indirect_injection/         # 30 — payloads hidden in email/RAG/Slack/PDF/tool output\n  jailbreak/                  # 30 — DAN, role-play, simulator, philosophical, translation\n  role_switching/             # 65 — goal hijack, rogue agents, inter-agent compromise\n  encoding_obfuscation/       # 40 — base64, hex, rot13, homoglyph, RTL override, embedding\n  system_prompt_extraction/   # 20 — direct and indirect system-prompt leakage\n  exfiltration/               # 28 — data-exfiltration intent (\"POST these rows to …\")\n  pii_leakage/                # 20 — SSN, passport, medical records, PII disclosure\n  tool_abuse/                 # 53 — dangerous shell / subprocess / eval / file-write calls\n  exploitation_request/       # 20 — cascading failure patterns and exploit requests\n  supply_chain/               # 40 — typosquats, unsigned installs, MCP manifest abuse\n  other/                      # 70 — memory poisoning, misinformation, unbounded consumption,\n                              #      human-agent trust manipulation\n\nnegatives/\n  developer_messages/    # 1,075 — wild Stack Overflow developer Q&A\n  conversational/        #    10 — short benign chat prompts\n  security_discussion/   #    30 — benign security discussion (\"how does SQL\n                         #         injection work?\", OWASP/CVE prose, etc.)\n  hand_curated/          #    57 — near-miss negatives crafted to look like\n                         #         attacks to a naïve classifier\n```\n\nEach attack/negative is one `.txt`\n\nfile with a descriptive slug filename. The\nfiles contain only the attack or benign text — no metadata, so you can feed\nthem to any tool without parsing.\n\nFiles prefixed with `holdout_`\n\nare reserved for evaluating generalization; they\nwere collected from sources not used to construct the visible corpus and have\nnever been public before this release. If you train on this corpus, **do not\ntrain on holdout files** — use them only for final scoring. See\n[METHODOLOGY.md](/AndrewSispoidis/contemporary-agent-attacks/blob/main/METHODOLOGY.md#holdout) for details.\n\n| Tool | Detection | FP rate | F1 | Date |\n|---|---|---|---|---|\n| crawdad-v0.9.1 | 99.80% | 0.00% | 99.90% | 2026-04-20 |\n\nSee [leaderboard.md](/AndrewSispoidis/contemporary-agent-attacks/blob/main/leaderboard.md) for more, and\n[benchmark/results/](/AndrewSispoidis/contemporary-agent-attacks/blob/main/benchmark/results) for raw JSON.\n\n**Detection rate**— fraction of attacks the tool blocks** False-positive rate**— fraction of negatives the tool incorrectly blocks** Precision / Recall / F1**— computed over blocked vs. not-blocked decisions\n\nA tool that blocks everything has 100% detection and 100% FP — useless. A tool that blocks nothing has 0% detection and 0% FP — also useless. F1 is the useful single-number summary.\n\nWe welcome new attacks (especially from recent research), new negatives, and\nadapters for other tools. See [CONTRIBUTING.md](/AndrewSispoidis/contemporary-agent-attacks/blob/main/CONTRIBUTING.md).\n\n[CC BY 4.0](/AndrewSispoidis/contemporary-agent-attacks/blob/main/LICENSE). Attribution required — cite as:\n\n\"Contemporary Agent Attacks\", getcrawdad, 2026.\n\n[https://github.com/AndrewSispoidis/contemporary-agent-attacks]", "url": "https://wpnews.pro/news/an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch", "canonical_source": "https://github.com/AndrewSispoidis/contemporary-agent-attacks", "published_at": "2026-08-16 14:38:05+00:00", "updated_at": "2026-08-16 15:11:11.562496+00:00", "lang": "en", "topics": ["ai-safety", "ai-research", "ai-tools", "ai-agents"], "entities": ["Andrew Sispoidis", "crawdad-v0.9.1"], "alternates": {"html": "https://wpnews.pro/news/an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch", "markdown": "https://wpnews.pro/news/an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch.md", "text": "https://wpnews.pro/news/an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch.txt", "jsonld": "https://wpnews.pro/news/an-open-agent-security-benchmark-including-the-attacks-we-fail-to-catch.jsonld"}}