In an AI security audit, the skeptic is the product

I wrote four security findings against a 47-line Node service. Two of them were false. One cites a file that does not exist.

Cloudflare's findings validator passed all four.

That is not a bug in the validator. It is the clearest way I found to show where the value of an AI security review actually sits.

cloudflare/security-audit-skill is a coding-agent skill that turns one agent into a coordinator of many: recon agents map the code, hunter agents look for bugs, and fresh verifier agents try to knock each finding down before it reaches a report. It was on GitHub trending this week with almost 2,000 stars in a day, 35,690 in total when I checked. MIT licensed. Cloudflare says it is the starting point of the vulnerability harness they now run across their whole fleet.

I measured it before reading it closely. The skill is 15 markdown files: 1,778 lines, 24,768 words, about 183 KB. At the usual four bytes per token that is roughly 46,000 tokens of instructions, loaded in pieces as a run needs them. Cloudflare's blog post says the first version was about 450 lines.

Two zero-dependency Node scripts sit beside the prose: validate-findings.cjs (773 lines) and validate-coverage-ledger.cjs (872 lines). Their own test suites ran clean on my machine, 34 and 31 tests.

The part that caught my eye is the proportion. Of the six phases, two exist only to disprove: Phase 3 hands every candidate to a fresh agent told to refute it, and Phase 5 sends another fresh agent over the final record (the quick profile merges the two). Each Phase 3 verifier prompt carries about 17.5 KB of fixed text before it sees the candidate: a six-step refutation prompt, a file-promotion procedure, and three schema branches copied in verbatim.

I did not install the skill, and no agent ran it. The README's install path is a skills CLI that writes into your agent's config, and I don't do that for a post.

Instead I wrote a small target, played the hunter by hand, and ran the skill's real validator on the result. The target is a notes service on node:http and node:sqlite with dummy users alice and bob. These are the two functions that matter:

function search(user, q) {
  const sql = "SELECT id, title FROM notes WHERE owner = '" + user + "' AND title LIKE '%" + q + "%'";
  return db.prepare(sql).all();
}

function getNote(user, id) {
  return db.prepare('SELECT id, title, body FROM notes WHERE id = ? AND owner = ?').get(id, user);
}

There is also an /admin/export route gated on X-Forwarded-For starting with 10..

Then I wrote four findings the way a hunter returns them, in the skill's schema:

search. Real. getNote, marked confirmed, with an observed result claiming it returned bob's note. False.findUser at src/users.js line 212. Invented: that file does not exist.

$ node validate-findings.cjs proposed.json
PASS: 4 findings valid

To be fair to it, it does enforce the contract. When I raised one record's overall severity above its impact, or put a severity on the needs_validation record, it failed, with exact paths. It checks shape, ordering, safe relative paths, and that a confirmed record has an observed result written in visible text.

It never opens a source file. It doesn't take the target path at all. The skill says this itself, in one line of Phase 4: validator success "proves format and ledger consistency only." A hunter's observed result is a string, and my invented one was schema-valid.

Next I wrote 15 lines of Node that open every cited file and check the line exists:

ok   admin/export-xff-gate
ok   notes/note-sql-concat
ok   notes/search-sql-concat
FAIL users/find-user-template-sql  src/routes/login.js: no such file; src/users.js: no such file

That kills the invented file. Cloudflare's harness has the same idea in plain code: per the blog, deterministic code confirms the cited files and paths exist before a finding moves on.

It does nothing for the getNote claim. Line 24 is real, and it really runs a query with the id in it. Every fact the record cites is true except the one that matters.

So I walked the Phase 3 prompt by hand. Step 2 says to rebuild the strongest control on the path. For getNote that takes one read: id and owner are bound as placeholders. Step 3 says to reproduce the minimum result yourself. My service is my own code with dummy data, so I ran it on loopback (the skill demands an OS-enforced sandbox before it runs anyone else's target; this was not that). Output trimmed:

search, normal     200 [{"id":1,"title":"groceries"}]
search, payload    200 [{"id":1,"title":"groceries"},{"id":2,"title":"payroll"}]
note, bob id       200 null
note, payload      200 null
note, payload 2    200 null
export, XFF 10.    200 [... both users' notes ...]

The search payload ' OR 1=1 -- hands alice the title of bob's note. Every getNote payload returns nothing. The export gate opens to a header alone, but whether a client can send that header in production depends on a proxy the repo doesn't contain, so it stays needs_validation with an owner check attached. That is the skill's rule for a deployment fact, and I agree with it.

The final file: one confirmed, one needs validation, two rejected.

$ node validate-findings.cjs findings.json
PASS: 4 findings valid

Same validator, same four fingerprints, same PASS. In between, two of the three "confirmed" bugs became rejected. Only reading the code at the cited line and running the payload told the files apart.

Finding candidates is cheap now. Cloudflare's blog reports 20,799 raw candidates from their discovery harness, of which about 12,057 survived validation, and an early validation rejection rate of 40% before better recon brought it down to 11%. At 40%, a team without a refuter gets four bad tickets in every ten. Somebody pays for those, and it's usually a maintainer.

That is also the theme of OpenAI's "Patch the Planet" post on coordinating disclosure as agents surface critical bugs, which was on the Hacker News front page the same day.

So my claim: when you judge an AI security tool, judge the step that tries to prove the finding wrong. The hunt is a hypothesis generator. The product is the record that survived.

Cost is real. In the Hacker News thread, jesse_dot_id reported at least 150k tokens on a fairly small FastAPI project before hitting a session limit. Issue #20 is a blind test on one seeded target, three rounds per arm, by a team with its own competing tool, which it discloses. There the quick profile cost a median $29.95 against $2.06 for one plain agent, and found the same share of the seeded bugs (a median recall of 0.467 for both). On a small, clean target the skeptic may have nothing to kill.

And a skeptic that shares the hunter's model is not fully independent. Per that same issue, the skill's verifiers inherit the session's model. Cloudflare's own pipeline triages with a different model from the one that hunts. Fresh context is not a different mind.

If you run AI reviews on your code today, what does the step that tries to disprove a finding look like: a second model, a test that must fail before the fix, or you?