In the first post in this series, Wade shared how our team laid the foundation for using AI in detection engineering. The short version is that an LLM doesn't know your stack. Without documentation and workflow context, it can produce plausible results that are difficult to trust, regardless of which model you use.
That post closed with an instruction: Validate before you ship, because the output can look completely right and still be wrong. Every detection an agent writes still needs to go through the same review and testing as a human-written detection.
Let’s discuss that last part, because it's harder than it sounds.
We'd built a detection-as-code pipeline, and we were proud of it. Rules live in a repository. Changes arrive as pull requests, and they get peer-reviewed, planned, and applied through CI. We had the structure of a software engineering team, and we tested queries against the logs we had. For everything else, we had never watched the detections fire. We shipped those because they looked right.
Validating a detection without the logs in your environment means running the attack technique yourself and watching it fire. We didn't have time to do that manually for every rule, and we had no automated way to do it.
So here's the question. How many of your deployed detections have you watched fire? Not reviewed – actually watched fire.
You don’t want to be answering that question the day an incident happens.
Now that detections are code, they need to be tested like code. This post is about what it took to give our detections tests. An AI model did one job to make that possible, which is picking the attack technique to run against each detection.
Before we get to the pipeline, it helps to see how a detection that looks right can still be broken. Reading a detection tells you what the query says, not what it actually does. It doesn't tell you whether the query matches any real logs, or whether the detection catches what you wrote it for. Here are five ways a detection can look right and still never fire:
That field isn't what you think it is. It might have been renamed between vendor versions. It might be nested a level deeper than the docs suggest. Or it might be sitting in the raw message instead of parsed into a queryable attribute. The query is valid, it's just pointing at something that isn't there.
Absence isn't emptiness. You write a condition for a field being missing, and the source sends it as an empty value instead of leaving it out. Depending on the query language, a wildcard matches that empty value, so the condition is never true and the rule never fires.
The query language doesn't behave the way you read it. A quoted phrase doesn't always match the exact string you typed. Wildcards work one way on a parsed attribute and another way in free text. Case sensitivity or one unescaped character drops your result set to nothing.
The data never reaches the query. The rule might be scoped to an index the logs don't route into. The source might have stopped shipping last week. Or an upstream parser change might have broken the field the rule depends on. The logic is fine. There's nothing for it to read.
The threshold is set past what your environment can produce. If the most you've ever seen is ten events in a window and the rule fires at a hundred, you've written documentation, not a detection.
Every one of these failures is silent. No error, nothing in the queue. A rule that can't fire looks exactly like a quiet environment.
That reality is why "we have N detections" was never a coverage number. That number tells you how many rules you've written, not how many attack techniques you've proven you'd catch.
These problems show up the first time you run the attack technique and check whether your detection fires.
We aren't a red team. I'm not a red teamer, and this isn't meant to replace one. Our approach starts from the detections we're writing or changing. For each one, AI picks the matching attack technique from a library of known techniques, the pipeline runs it, and then checks whether the detection’s query caught it. A red team tells you what you missed. This approach tests whether the detection you wrote works.
This post also isn't an implementation guide. There's no code to copy. The emulation tooling we build on is open source, but the pipeline around it is our own. Instead, it covers why we built this, how the pieces fit together, and what we've learned running it. If you want the technical details, I'll be walking through them in a session at Datadog Detect in Chicago on November 2.
Detection-as-code is half of what makes this possible. The rules being tested are code. So is the thing doing the testing.
Our attack library is a fork of Stratus Red Team, the open-source cloud adversary emulation project maintained by Datadog. Every technique is defined in that fork, released as pinned versions, and changed through pull requests like anything else.
Because the catalog is code, the model mapping a rule to an attack technique gets a fixed list to choose from. Anything outside that list simply can't run. And if nothing in the catalog fits, the answer is no mapping, which is a useful answer in its own right.
We fork instead of consuming upstream so we can add techniques for the behaviors we care about and control when the catalog changes. That gives us a versioned test input rather than an external dependency that can change without notice.
A fork can also go stale. Keeping it synced is ongoing work, and if you skip it, your catalog quietly stops reflecting what techniques attackers actually use. That's part of the price.
Our detections are code, and our attack library is code, which is the whole reason any of this can run in CI. You need both. Without an attack library, there's nothing to test your detections with. And if your detections live in a UI instead of code, there's nothing for CI to run the attack library against.
The AI model has one job in this pipeline: mapping. It reads the whole detection, including the query, the MITRE ATT&CK tags, and the description, and picks the attack technique from our library that would trigger it.
That's the hard part. The library labels each attack technique with a MITRE ATT&CK tactic, not a technique ID, so a detection's ATT&CK tag can't be matched against it directly. The query does most of the work, especially the event names it looks for, backed by a reference we wrote for each platform that maps common events to attack techniques. The tags and description add what the author meant to catch. It still takes judgment, and doing it by hand across hundreds of detections doesn't scale.
Everything else is plain scripts. Running the attack technique, checking the SIEM, and posting the result on the pull request don't need judgment, so there's no reason to put a model where a script belongs.
We also put guardrails on the model's answer. It has to reply in a fixed format that includes how confident it is, and it's told to return nothing rather than guess. A low-confidence answer gets treated the same as no answer, so a shaky guess never runs.
Beyond the guardrails, we want to baseline the mapping against open-weight models. We'll give them the same detections we map today and see if they pick the same attack techniques, or if one model keeps favoring certain techniques or platforms. We also want to see whether a model's confidence score matches how often its picks actually pass. That tells us whether our results come from the context we built or from the model we happened to pick.
When a pull request adds a detection, changes its query, or turns a disabled one back on, CI kicks off. Re-enabling counts because a rule that's been off is the one most likely to have gone stale. Edits that only touch tags, descriptions, or other metadata get skipped, since the logic didn't change. From there:
The model maps the rule to an attack technique.
The pipeline runs it in an isolated cloud environment with credentials pulled at runtime from our 1Password Connect server.
A bot posts a pass or fail table back on the pull request.
The attack technique runs for real. Nothing is replayed, and nothing is synthetic.
What matters most is that your change isn't deployed yet. The pull request hasn't merged, so the SIEM is still running the old version of a rule you're modifying, and a rule you're creating isn't there at all. Either way, there's no signal from your version to wait for. So the pipeline runs the attack technique and then searches the logs with your query. That shows the events your rule is looking for actually landed, in the shape you expected, from a real attack technique.
“Your logic matches reality" and "your detection works" are both worth having, but they aren't the same claim.
Two details matter once you're running it.
Threatest, an open-source detection testing framework also maintained by Datadog, runs the attack techniques from our library. Threatest can also check whether the SIEM caught one. We customized that check for our pipeline.
Checking that matching logs showed up isn't enough on its own. Your query might match an earlier test, or other activity that happens to look the same. So every time the pipeline runs an attack technique, it stamps a unique ID into it, and validation looks for that ID in the logs your query returns.
Not every green check proves the same thing. There are two outcomes.
The model mapped a technique. The pipeline ran the attack technique and found its events, with that run's unique ID in them, using the rule's own query. That's the strongest result a pull request can give you. The logic matches a real attack technique.
Mapping takes judgment, so the model can still pick the wrong attack technique. When it does, your query won't find that technique's events and the check fails. In our system, a wrong pick shows up as a failure, not a false pass. The attack technique either produced the events your query looks for or it didn't.
Nothing mapped. There's no attack technique to run, so the pipeline searches the last 30 days of logs with each of the rule's queries. If every query the rule needs matches at least once, it passes. That tells you the logic matches something real. It doesn't tell you it fires on the right thing.
The split exists because cloud platforms can be attacked on demand and most SaaS sources can't. You can't safely make a third party's audit log produce an arbitrary event, so most SaaS rules never get a technique.
SaaS is most of what most teams monitor, so most rules land in the second outcome. That's still better than shipping on "it looks right," but it's worth knowing which check you got.
A green check that means "this could match" isn't the same as one that means "** this matched an attack technique the pipeline just ran.**"
Some rules don't get a check at all, and that's a limitation we’ve accepted. Anomaly, new-value, and correlation rules get skipped. The first two need a baseline that one attack technique can't build, and correlation rules depend on other rules firing first.
The check is advisory. A failed check shows up on the pull request, but it won't stop the merge. A CI error or an expired cloud credential shouldn't be able to keep a detection from shipping. The tradeoff is that nothing forces anyone to act on a failure, so we read every result as part of review.
When a check fails, we don't merge until we know why. It's often one of the problems from earlier in this post, like a field that isn't where the query expects it or data that never reaches the index. The author pushes a fix to the same pull request, and the check runs again. If the model picked the wrong attack technique, the result shows that too, because the table names the technique it ran.
It only runs on pull requests. A detection gets tested when someone changes its query or turns it back on, and deployed detections that nobody touches never get retested. Eventually, we'd like a way to kick off a run on demand, without opening a pull request.
Reviewers look at evidence instead of judging a query by eye, and "did you test this?" stopped being a question anyone had to ask.
Running an attack technique and checking a rule against it used to take hours by hand: setting up the environment, running the technique, waiting for the logs to land, and searching for them. Now it takes minutes, on every pull request that adds a detection or changes an existing one's query.
Most of those time savings come from cloud rules. SaaS rules are still checked only against existing logs, but that doesn't have to be permanent. The attack techniques in our library are API calls against a cloud provider, and SaaS products have APIs too. In an isolated test tenant, you can call those APIs to produce the same audit events an attacker would, and turn that into an attack technique.
Building one takes context about the product, like which API calls produce which events, and what those events look like when they land. It's the same point Wade made in part 1. A model is only as good as the context you give it. SaaS coverage remains an open area for this approach. For 1Password, the logical place to start is our own product, where we can map API calls to the audit events they generate.
Take one deployed cloud detection, run the attack technique by hand once, and watch whether the signal fires.
Either your detection works, or you just found out it never did. Better to find that out now than during an incident.
If it didn't fire, it's probably not the only one. That's when automating this starts to pay off. The 1Password CLI can reference approved secrets from scripts and automation without pasting plaintext credentials into prompts or source code. Connect secure secret management to detection engineering workflows with our free developer tools.