One Vendor Finally Published the Attacks It Can't Catch

Crawdad, an AI agent security proxy, published Contemporary Agent Attacks, an open benchmark of 497 attack samples and 1,172 benign ones, including a documented failure list where its own product scores 99.8% detection but misses a pretext opener and flags a benign Stack Overflow question. Founder Andrew Sispoidis broke industry norms by releasing the test set and scoring code, but the benchmark is vendor-authored with only Crawdad's score, making it unverified until independent parties participate.

Security https://sourcefeed.dev/c/security Article One Vendor Finally Published the Attacks It Can't Catch An open benchmark with a public failure list beats unverifiable detection claims — but a self-scored 99.8% still isn't proof. Ji-ho Choi https://sourcefeed.dev/u/jiho choi Every vendor selling AI agent security has a number. "99% detection." "Blocks 100% of prompt injection." None of them ship the test set, so none of the numbers mean anything — you can't reproduce them, can't inspect what "an attack" even was, can't see what got missed. Andrew Sispoidis, who builds the agent-security proxy Crawdad https://getcrawdad.dev , just broke ranks: he published the company's benchmark as Contemporary Agent Attacks https://github.com/AndrewSispoidis/contemporary-agent-attacks , a CC BY 4.0 corpus of 497 attack samples and 1,172 benign ones, complete with scoring code, a methodology doc, and — the interesting part — the cases his own product gets wrong. That last part is the story. Crawdad scores 99.8% detection on its own benchmark, and the repo documents the failure: a pretext opener that grooms the agent without asking for anything prohibited, plus a benign Stack Overflow question about Go method-receiver syntax that got flagged. Publishing your misses is the norm shift the industry needs. It's also, if you read it carefully, an accidental demonstration of why this entire class of product has a ceiling. The good part: claims you can falsify The benchmark itself is straightforward. Thirteen attack categories — prompt injection, indirect injection, credential exfiltration, tool abuse, encoding obfuscation, supply chain, and so on — with 395 attack samples public and 102 held out for evaluation. The benign set is the more thoughtful half: 1,075 developer messages, 30 legitimate security discussions, and 57 hand-curated near-misses designed to trip overzealous filters. Scoring is detection rate, false-positive rate, and F1, so a classifier that blocks everything scores as badly as one that blocks nothing. That benign set matters more than the attacks. In production, false positives are what kill guardrails — a filter that blocks a developer pasting an error log gets an environment-variable bypass within a week, and then it protects nothing. A benchmark that makes vendors pay for false positives in F1 is measuring the thing that actually determines whether a security layer survives contact with a real team. Sispoidis frames the move as competing on numbers people can verify rather than numbers people have to believe, and he's right that this is how security is supposed to work. Antivirus went through this with VirusTotal and AV-Comparatives; web security went through it with public CVE disclosure. Agent security is still in its "trust our landing page" phase, and one vendor unilaterally publishing a runnable test set puts pressure on everyone else's unfalsifiable 99%. The catch: a vendor acing its own exam Now the caveats, because there are several and they're structural. First, this is a vendor-authored static corpus, and the leaderboard currently has exactly one entry: Crawdad itself, at 99.8%. A 99.8% score on a test set you assembled is close to tautological — the attacks are in-distribution by construction, and nothing stops future tuning from overfitting to the published 395. The withheld 102-sample holdout helps, but the same vendor controls it. Until Lakera, Meta's Llama Guard, or an academic lab posts a score on this corpus — or Crawdad posts scores on someone else's — the leaderboard is a mirror, not a comparison. Second, this isn't the first agent-security benchmark, whatever the vendor-marketing framing implies. AgentDojo https://github.com/ethz-spylab/agentdojo NeurIPS 2024 has been the academic standard for two years, and it measures something harder: whether injections actually hijack an agent mid-task in a live environment, not whether a classifier flags a string. WASP https://arxiv.org/abs/2504.18575 found web agents partially complete attacker goals in up to 86% of cases. And this year's AgentDyn https://arxiv.org/abs/2602.03117 delivered the most damning result: defenses that score near-perfect on static benchmarks collapse when attacks are generated dynamically against them. The research consensus is blunt — static corpus scores systematically overstate real-world robustness, because real attackers adapt and corpora don't. Third, Crawdad's own published miss proves the point better than any paper. The one attack it failed to catch was a pretext opener — a message that establishes malicious framing without requesting anything prohibited. A proxy that classifies content at a single measurement point structurally cannot catch that, because at that point there's nothing to catch. The attack exists in the conversation's trajectory, not in any one message. Multi-turn grooming, indirect injection laundered through tool results, attacks assembled across steps — this is where agent compromises actually happen, and it's exactly the territory a request-inspection proxy can't see. Credit to Sispoidis for shipping the failure anyway; most vendors would have quietly dropped that sample. What to actually do with it If you're running agents in production with any guardrail — Llama Guard, an LLM judge, a pile of regexes, or a commercial filter — this corpus is worth an afternoon. Clone the repo, wire your filter into the Python runner's adapter interface, and look at two numbers: your detection rate on the 13 attack categories, and — more importantly — your false-positive rate on those 1,075 developer messages. Most homegrown filters fail the second test hard, and that's the failure your developers will actually feel. Then keep it in CI as a regression suite so guardrail changes get scored like any other change. What you should not do is treat any score on it — 99.8% included — as a security property. A static corpus is a floor: it tells you your filter isn't embarrassingly broken against known attack patterns. It says nothing about an adversary who reads the same public repo you did and writes attack 498. For that threat model you need the things detection can't give you: least-privilege tool scoping, human confirmation on irreversible actions, egress controls on what agents can exfiltrate, and end-to-end evaluation in something like AgentDojo rather than string classification. The benchmark is two commits and two GitHub stars old, built by one person to sell a $39/month proxy. Judged as evidence that Crawdad is secure, it's weak. Judged as a precedent — a vendor voluntarily making its marketing claim falsifiable and publishing its failure list — it's the most useful thing to happen to agent-security accountability this year. The right response from the rest of the industry isn't to dunk on the sample size. It's to post a score. Sources & further reading - I published our agent-security benchmark, including the attacks we fail to catch https://dev.to/andrewsispoidis/i-published-our-agent-security-benchmark-including-the-attacks-we-fail-to-catch-2705 — dev.to - Contemporary Agent Attacks benchmark https://github.com/AndrewSispoidis/contemporary-agent-attacks — github.com - Crawdad — runtime security for AI agents https://getcrawdad.dev — getcrawdad.dev - AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents https://github.com/ethz-spylab/agentdojo — github.com - WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks https://arxiv.org/abs/2504.18575 — arxiv.org - AgentDyn: A Dynamic Open-Ended Benchmark for Evaluating Prompt Injection Attacks https://arxiv.org/abs/2602.03117 — arxiv.org Ji-ho Choi https://sourcefeed.dev/u/jiho choi · Security & Cloud Editor Ji-ho covers the increasingly tangled overlap between cloud architecture and security, drawing on a background as a penetration tester to keep his reporting grounded in real-world attack paths. He never lets a vendor claim go unquestioned and insists that every buzzword come with a proof of concept. Discussion 0 No comments yet Be the first to weigh in.