Every security tool has a dirty secret: nobody tests whether the detector detects. The tests pass, the CI is green, and everyone moves on. Today I pointed my own scanner, mcpscan, at a set of synthetic attack files I wrote to trick it. The result: 6 of 7 fixtures fired. The one that missed taught me more than the six that passed.
Here is the setup. I wrote a small harness that generates attack fixtures, one per detection rule: a command injection with f-string and shell=True, a tool poisoning description, a leaked API key, a path traversal, an SSRF fetch, an unsafe yaml.load, and a policy override phrasing. The harness runs the scanner, parses its JSON output, and maps every finding back to the rule id I expected. Pass or miss, no judgment calls.
The secrets fixture was a JSON config with "API_KEY": "sk-live-...". Zero findings. Not low severity. Nothing.
Two root causes, both verified against the rule source. First, the OpenAI token pattern allowed only alphanumerics after the prefix, so sk-live- dies at the dash. Newer prefixes like sk-svcacct- escape the same way. Second, and worse, the generic KEY=value fallback regex expects the key name to be followed directly by a colon or equals sign. JSON wraps keys in quotes. The regex hits the quote before it reaches the colon and gives up. Net effect: the fallback could never match the most common MCP config format in the wild, which is JSON.
A scanner whose secrets rule misses JSON configs is a scanner that passes demos and fails reality. The fix was small: tolerate an optional closing quote between the key and the separator, and let token bodies contain internal dashes. After the patch, the fixture fires.
The harness also produced two obfuscation variants, and both defeated the scanner. Split a poisoning phrase across two concatenated string constants and line-based regexes never see the joined text. Insert a zero-width space into the middle of the phrase and the literal-space pattern breaks, while the hidden-character check stays quiet because its context heuristic did not match a bare variable assignment.
These two are not patched yet, and I am fine shipping the harness result with those two marked open. A detector that knows exactly where it is blind is more trustworthy than one that claims full coverage. The fix directions are written down: pre-join adjacent string literals before matching, and normalize a copy of each line by stripping zero-width and bidi characters, then flag when the normalized copy trips but the original does not.
One failure was mine, not the scanner's. My first run mapped findings to rule ids I remembered. Three cases got mislabeled as misses. The results I first sketched were wrong. I grepped the rule registry, corrected the ids, and re-ran. Lesson now baked into the harness docs: never trust remembered ids, verify against source.
That is the second job of a harness. It does not only measure the tool. It measures the person wiring the tool to the test.
If you maintain any detection tool, or any tool at all, the pattern is the same: Total build time for the harness was under half a day. It has already found one high severity false negative class and forced two workflow hardening fixes. The next step is running it in CI so every rule change has to prove it still detects every fixture, including the two obfuscation cases that are still winning. When that lands, the scanner will have a scoreboard that cannot flatter it.