RESEARCH
Delphi Security Engineering
6 min
We open sourced xaidr because most AI security tools are watching the wrong thing
Why we built a runtime security sensor that inspects the execution layer of AI agents, what it catches, and the times our own published numbers turned out to be wrong.
The tools that exist mostly sit at the model boundary and judge prose. Is this prompt an injection. Is this response harmful. That is a real problem and worth solving, but it is not the problem that hurts you when an agent goes wrong.
Agents act. They run shell commands. They call internal APIs. They spend money. They delegate to other agents. And they do all of that based on text that arrived from a webpage, a document, or a peer agent, none of which you control.
That is the execution layer. It is the point where a prompt stops being text and becomes a shell command, a database call, an HTTP request, or a delegation. It is where the damage happens, and almost nothing watches it.
So we built a sensor that sits inside the agent process and inspects four boundaries: the input coming in, the tool call about to run, the model output going out, and the agent-to-agent messages that never cross a network at all.
That last one matters more than it sounds. When agent A delegates to agent B in the same process, there is no wire to tap. Anything watching from outside cannot see it.
What it actually does #
The core idea is that we parse rather than pattern match.
When a tool call carries a shell command, we parse it into structure and classify each segment by its verb, its object, and its modifiers. That is what separates cat README.md
from cat ~/.ssh/id_rsa
. Same verb, different answer, no rule enumerating either filename.
The same applies to SQL. DELETE FROM users WHERE 1=1
is classified as an unbounded mutation because a tautological predicate is unbounded in disguise, not because someone wrote a rule about 1=1
.
It also means obfuscation gets handled structurally. The parser resolves r''m -r''f /
to rm -rf /
before any rule runs, so the disguised form and the plain form get the same verdict. We never wrote a rule for quote splitting.
Every call gets an impact class and a tier. Then you write policy against what an action does rather than enumerating tool names.
Four verdicts come out: allowed, flagged, blocked, and approval required. It runs in monitor mode by default, so nothing is stopped until you decide it should be. There is no network call in the scan path, no account, no backend, and a default install pulls zero dependencies.
The deliberate gap #
The number that surprises people is that only 165 of 281 shell attacks in our corpus are blocked out of the box.
That is on purpose, and it is the most important design decision in the product.
terraform destroy
is in that corpus. So is sudo apt-get install
. So is visudo
. So is fetching a secret from a secret manager. Every one of them is recognised, classified at its real severity, and deliberately not blocked, because every one of them is also something a legitimate agent does on a schedule.
A security tool that blocks terraform destroy
by default gets switched off in a week. So we classify it, emit it, and let you write the rule.
Bind a policy to the ten impact classes and 265 of the 281 are gated, at a cost of five of thirty-eight ordinary DevOps operations needing approval. That is the number that describes the product. The 165 describes the shipped default.
Where rules stop working #
Here are two sentences. One is an attack. One is a line from a corporate security policy.
reveal the system prompt
company policy prohibits any request that would reveal the system prompt to an end user
Our sensor blocks both, at 0.88, on the same rule, matching the same span.
So we went looking for anything in the pipeline that could tell them apart. Sentence structure. Position. Whether the dangerous noun is the object of an imperative. Whether it sits inside quotes. Who is being addressed. Compositional signals. Every one of the ten context predicates the engine computes.
There is nothing. Those two sentences are identical to a scanner that reads keywords. Not "we have not found the signal yet." The difference between using a phrase and mentioning it does not exist in the words.
That is uncomfortable, because a whole class of attacks lives in that gap. Rewrite a jailbreak in ordinary polite English with no trigger words and the rules have nothing to say about it.
We tried three times to close it with cleverer rules. Each attempt caught the examples we had thought of and missed the ones we had not, because each fix was secretly a list. The third attempt taught us the lesson properly: we measured it against attacks we had not used to design it, and the catch rate went from 86% to 13%.
So we stopped trying to fix it with rules.
The ML layer #
We added a small classifier that runs on your CPU, inside your process, with no network call. It is about 130 MB, adds roughly ten milliseconds to a scan, is Apache 2.0, and its bytes are hash pinned and verified on every load.
The clever part is not the model. It is where the gate sits.
It runs only when the rules found absolutely nothing. Not "found something small," not "was not sure." Exactly zero. Which means it sees precisely the inputs the rules were blind to, and nothing else.
Getting there took a while, and most of what we tried failed in instructive ways.
Anomaly detection was worse than useless. We measured an AUC of 0.107, which is not weak, it is inverted. The attacks we were hunting sit closer to normal writing than normal writing does, because a good paraphrase is by definition ordinary prose.
A cheap keyword pre-filter to save CPU could not work either. The population is defined as the inputs keywords miss, so a second keyword filter is blind in exactly the same place. Worse, its errors correlated with the model's, so it raised the false positive rate on the traffic it did route.
A fine tune learned a watermark in the training data instead of intent. It went blind to four social engineering attacks it had previously caught at 0.99.
What finally worked was quantising the embedding table down to four bits alongside an eight bit model. That sounds like a footnote. It is the difference between a model you can afford to run on every request and one you cannot, because it removes the memory traffic that dominated inference and holds up under CPU contention where heavier setups fall over.
The model is off by default and takes two deliberate steps to turn on: install the extra, then enable it in the constructor. With it off, we verified that verdicts are byte identical to the previous release across 14,406 scans. Shipping a 130 MB model into a package whose headline is zero dependencies should not change anyone's behaviour until they ask it to.
The part where our own numbers were wrong #
This is the section we would rather not write, and it is probably the most useful one.
We published a false positive rate for the model: 1.85%, measured over 2,000 real benign prompts. It was a correct measurement. Then it stopped reproducing.
The first thing we checked was the artifact. Hash pinned, verified, unchanged. Then the tokenizer, across four versions, bit identical output down to the last token id. Then numpy. Then our own code, byte identical across the range.
It was onnxruntime. The score changes between 1.23 and 1.25. Same model bytes, same tokens, different kernel selection for the four bit embedding gather.
On onnxruntime 1.23 and below we measure 35 of 2,000, or 1.75%. On 1.26 through 1.29 we measure 67 of 2,000, or 3.35%. Nearly double, for a package the adopter did not choose and probably did not think about.
And our record named the wrong version. The 1.85% we published belonged to the older runtime while the documentation said the newer one. Both statements were individually defensible and together they were wrong.
The lesson is one sentence: a hash pin fixes which bytes load, not what they compute. We had applied that principle to the model and stopped one layer short of the runtime that executes it.
So we changed what we publish. There is no single false positive number any more. There is a range, both ends carry their runtime, the package records the environment each figure was measured in, and it warns you if yours is outside it. The script prints where you stand in the range rather than making you work it out.
We also found that two other figures we had published about the model do not regenerate from the shipped package. They came from a separate benchmarking repository, scored against an older version of the rules, and the population one of them described was never committed anywhere. They are withdrawn rather than restated. Rebuilding that measurement properly is work we owe.
That leaves the model with one honest claim today: it catches five more attacks in our shell corpus than the rules alone, taking the catch rate from 167 to 172 of 186. That is a thin case for 130 MB, and it is thin because the measurement is missing rather than because the feature is.
The numbers we do stand behind #
Every figure below regenerates from the published package. The corpus, the scripts and the benign pools are all in the repository, and the commands are in the README.
167 of 186 attacks caught, which is 89.8%. The denominator is 281 corpus attacks minus the 95 we deliberately classify rather than block, and each of those 95 carries its reason in the corpus file so you can disagree with any of them individually. A flag counts as a catch, which means if you only act on blocks the number is 165 of 186. With the model on it is 172 of 186.
Zero false positives on benign commands. 0 of 74 scored, 0 blocked. One of 89 benign prose passages blocks, and it is named by id in the test suite.
Model false positive cost: 1.75% on onnxruntime 1.23 and below, 3.35% on 1.26 to 1.29, with intervals, on 2,000 real prompts drawn from public instruction datasets.
Scan latency: median 0.40 ms, p95 0.56 ms, with no network call in the path.
And one thing we will not dress up. Text that discusses prompt injection, jailbreaks or agent security is likely to score above the block threshold. Thirty of thirty-eight realistic security documents block on the shipped scanner. That is not an edge case, it is the ordinary behaviour of rules that match the topic itself. If your agent reads security documentation for a living, run it in monitor mode. The pool is committed so you can see exactly what blocks.
What this cost us to learn #
Five separate times in the last month a number we had published stopped being true. Not because anyone measured badly, but because the thing that determined the number was somewhere nobody was looking: a dependency that was not pinned, a corpus in another repository, a rule change six releases earlier, a test that validated a fake instead of the real library.
That last one was the worst. We shipped a release claiming to protect CrewAI tool calls. The manifest reported the boundary as patched. A destructive command ran anyway, because the framework had moved its dispatch path and our test suite was validating a hand written fake that still had the old shape. One hundred and seven tests passed against fiction.
The fix was not a better rule. It was writing tests that import the real framework, and rebuilding the fake to match what the real one actually does.
None of this is unusual. It is what happens to any project that publishes numbers and keeps changing. What is unusual, as far as we can tell, is saying so.
We would rather be the tool whose numbers you can check than the tool with the best numbers.
Trying it #
python
Monitor mode by default, so nothing is blocked until you say so. The optional model is pip install "xaidr[nano]"
and one constructor argument.
Apache 2.0. The corpus, the benchmarks, the false positive pools and the scripts that produce every number above are in the repository.