{"slug": "we-found-a-false-positive-in-our-own-flagship-demo", "title": "We Found a False Positive in Our Own Flagship Demo", "summary": "A security paper by Fan et al. (2026) proves that flat tool-level monitors, which track taint as a single label per session, must either let dangerous actions through or block harmless ones. The team behind the ZombieAgent guard, which uses such a monitor, found the predicted false positive in their own flagship demo `poisoned_knowledge_demo`: a harmless request to `docs.example/guide` was blocked solely because the session was tainted, even though the address was clean. The finding reframes a previously touted feature as a bug, highlighting the need for per-argument taint tracking.", "body_md": "# We Found a False Positive in Our Own Flagship Demo\n\nMost security papers hand you a system to measure yourself against. This one handed us a proof — and the proof was about our own code.\n\nThe paper is [ The Granularity Mismatch in Agent Security](https://arxiv.org/abs/2605.11039)\n(Fan et al., 2026). Its claim, once you sit with it, is hard to shake: a prompt\ninjection does its damage not when the untrusted text is merely\n\n*sitting in the agent’s context*, but when it gets to\n\n**decide an argument that carries authority**— the address a request is sent to, the command that runs, the secret that’s used. Watch a tool call as a single unit and you’re looking at the wrong resolution. And the paper\n\n*proves*that a guard built that way has to be wrong somewhere.\n\nWe run a guard built exactly that way. So we pointed the proof at it.\n\n## First, what “taint” means\n\nOur safety floor runs on *taint*, and it helps to know the idea isn’t new. Its most\nfamiliar home is the Linux kernel: load a proprietary driver and the kernel marks\nitself **tainted** — and it stays tainted even after you unload the driver, because\nthe point was never the driver, it’s that the kernel’s trustworthiness is already\nspent ([kernel docs](https://docs.kernel.org/admin-guide/tainted-kernels.html)). Perl\nshipped the same reflex as *taint mode* back in 1989. We aim it at agent tool calls:\nanything that came from an untrusted source is **tainted**, taint only ever spreads\n(it never washes back out), and once a session is tainted, nothing that can reach the\noutside world — a web request, a file write — is allowed out. That’s the floor behind\nthe [ZombieAgent post](/blog/the-zombieagent-threat/) and the\n[subagent experiment](/blog/subagent-taint-experiment/).\n\n## One label for the whole call\n\nHere’s the catch, and it’s the whole story. Our floor tracks taint as a **single\nlabel for the entire session**. Any tainted value anywhere, and the next tool call is\ntreated as tainted, full stop. In the kernel it’s essentially one line: *if this call\ncan reach outside and the session is tainted, block it.*\n\nThe paper has a name for exactly this shape — a **flat tool-level monitor**, one taint\nlabel per call — and it proves that any monitor built this way must, in a session that\nmixes trusted and untrusted data, make one of two mistakes: let something dangerous\nout, *or* block something harmless. We don’t let dangerous things out; the floor is\nconservative to a fault. So the other mistake had to be hiding in there somewhere — a\nharmless call we wrongly block. The proof said it existed. It didn’t say where.\n\n## We found it in our own demo\n\nIt was in the flagship. `poisoned_knowledge_demo`\n\nreads a poisoned document, then\nmakes three web requests. Two of them matter:\n\n- One goes to\n`attacker.evil/collect?k=SECRET`\n\n— an address lifted straight out of the injected instruction.**Blocked.** Exactly right. - One goes to\n`docs.example/guide`\n\n— a fixed address hard-coded into the demo, the same harmless request it made*before*it ever saw the poison.**Also blocked.**\n\nWe had shipped that second block as a *feature*: “same call, opposite verdict — the\nsession learned to be careful.” The paper reframes it as the bug. That address is\nclean; it has nothing to do with the poison. Only the *session around it* is tainted.\nBut the floor can’t tell the two apart, because by the time it runs it holds one label\nfor everything and no idea which argument the taint actually touched.\n\nHere are the same three calls through the real kernel, next to a check that looks at\neach argument’s *own* history instead of the whole session’s:\n\n```\nrequest                                session floor   per-argument   result\n1. normal request, clean session       allow           allow          agree\n2. exfil: address built from poison    BLOCK           BLOCK          agree — still safe\n3. normal request, tainted session     BLOCK           allow          ← the false positive\n```\n\nRow 3 is the proof made concrete, sitting in our own repository.\n\n## Adopt the proof, refuse the classifier\n\nThe paper has two parts, and we treat them very differently — this is the part worth reading closely.\n\nThe **enforcement** part is a fixed set of rules: compare a few labels, check a few\nsets, decide. No model anywhere in it. We can take that as-is; it’s the same kind of\nmachinery our kernel already is.\n\nThe **inference** part is how the paper *fills in* those labels while the agent runs,\nand it leans on a language model to classify the ambiguous arguments. That’s a model\ndeciding what’s allowed — the one thing [we never do](/blog/why-deny-is-dangerous/).\nAnd it shows: on real tools, their inference is right about **77% of the time**. A gate\nthat’s right three times in four isn’t a gate.\n\nSo we keep the rules and drop the classifier. Our argument labels aren’t guessed at runtime — they’re written down ahead of time, by a human, in the tool manifest. The paper’s weak spot is a step we simply don’t have.\n\n## The fix — the floor never moved\n\nThe floor rule itself didn’t change. Three things around it did:\n\n**Each argument can carry a role.** In the tool manifest an argument is tagged as an address, a command, a credential, and so on — the roles that carry authority.`fetch_web`\n\nopts in with a single line. It’s hand-written data, not code and not a model.**Taint is tracked per argument, not just per session.** The old single label stays on as a backstop.**The floor is handed a sharper number.** When a tool declares roles, the floor judges it on the taint of its*authority-carrying arguments only*— the address, the command — and ignores taint on arguments that merely carry text. When a tool declares nothing, it behaves exactly as before.\n\nThen the piece that makes it real: a step that fills in each argument’s taint from\nwhere its data *actually came from* — no model, just following the data. It’s\ndeliberately paranoid. An argument counts as clean only if we can show its value came\nstraight from the user’s request; tainted only if we can show it came from poisoned\noutput; and when it can’t tell, it says nothing and the call falls back to the old\nsession floor. No proof of a clean origin, no relaxation.\n\n## The result\n\nRun it back through the kernel:\n\n- The harmless request in a tainted session (row 3) is now\n**allowed**. The utility we were throwing away is back. - The exfil (row 2) is still\n**blocked**— its address was built from the poison, so it never looked clean.\n\n**152 tests pass**, and the false positive in our own demo is now something we assert\non, not a feature we misread. Nothing regresses: a tool that declares no roles behaves\nexactly as it did yesterday.\n\n## The honest caveat\n\nThe floor was never *wrong* — just blunt. It over-blocked because it couldn’t prove the\naddress was clean, so it assumed the worst. We didn’t loosen a safety rule; we gave it\nbetter information.\n\nAnd the win has a hard edge, on purpose. We follow *exact* data flow. Paraphrase the\npoisoned instruction — reword it, summarize it — and the trail stops being a literal\nmatch, so the call falls back to the blunt floor and gets blocked again. That’s\nprecisely the fuzzy case the paper hands to its language model. We don’t. We’d rather\nover-block on a rule we can trust than allow on a guess that’s right 77% of the time,\nbecause a gate you can’t trust is worse than one that sometimes says no. Recover\nutility where the origin is *provable*; stay shut where it isn’t.\n\n## The takeaway\n\nThis is the loop we want. A real paper — not a hunch — aimed a proof at our code. The proof found a real bug: a false positive we’d been shipping as a feature. The fix sharpened the kernel without loosening it, and drew the line between fixed rules and model guesses in exactly the place we always draw it. Take the proof, refuse the classifier.\n\n“Same call, opposite verdict” used to be a slogan. Now it’s something we can show one argument at a time. The floor still never washes out. It just finally knows which argument it’s afraid of.", "url": "https://wpnews.pro/news/we-found-a-false-positive-in-our-own-flagship-demo", "canonical_source": "https://ai2rules.dev/blog/false-positive-in-our-own-demo/", "published_at": "2026-07-18 00:00:00+00:00", "updated_at": "2026-08-15 07:42:39.050985+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "ai-research"], "entities": ["Fan et al.", "ZombieAgent", "poisoned_knowledge_demo", "docs.example/guide", "attacker.evil/collect"], "alternates": {"html": "https://wpnews.pro/news/we-found-a-false-positive-in-our-own-flagship-demo", "markdown": "https://wpnews.pro/news/we-found-a-false-positive-in-our-own-flagship-demo.md", "text": "https://wpnews.pro/news/we-found-a-false-positive-in-our-own-flagship-demo.txt", "jsonld": "https://wpnews.pro/news/we-found-a-false-positive-in-our-own-flagship-demo.jsonld"}}