cd /news/ai-agents/i-benchmarked-a-security-audit-skill… · home › topics › ai-agents › article
[ARTICLE · art-170462] src=dev.to ↗ pub= topic=ai-agents verified=true sentiment=· neutral

I benchmarked a security-audit skill on Claude, Codex and Antigravity. Here are the numbers.

A developer built two open-source Apache-2.0 security-audit skills for AI coding agents and benchmarked them against OWASP Juice Shop's 32 line-marked vulnerabilities, finding Claude (Opus) caught 23/32 with exact file and line, Codex 18/32, and Antigravity (Gemini) 6/32. Earlier runs on an older Claude model scored only 7-8/32 because the agent logged issues as "noted" without promoting them to findings; a deterministic pre-pass requiring each dangerous-code category to be closed with a cited finding or dismissal raised scores to 12-13/32 on the same model. All nine vulnerabilities missed in Claude's best run were also missed by Codex and Antigravity, and the developer notes Antigravity stops early, an unresolved issue on the roadmap.

by read5 min views4 publishedNov 16, 2026

Most "AI security audit" prompts I have seen share the same problem: the report looks great and you cannot tell what was actually checked. An agent that writes "authentication: verified" after reading one file is worse than no audit, because someone will set a budget on it.

I work with Italian public administrations and non-profits, so I built two open-source skills for AI coding agents that try to fix this:

Both are Apache-2.0, work in English and Italian, and run on Claude Code, Codex and Antigravity. This post is about how I measured them, including the results that are not flattering.

check_delivery.py fails the audit if a dangerous-code category found by the pre-pass is left without a verdict, if forbidden words like "compliant" appear, or if secret values leak into the folder. My own test apps have a weakness: I wrote the bugs and I wrote the checklist, so they measure obedience more than generalization. Juice Shop is better: its authors mark the vulnerable lines of their coding challenges in the source (vuln-code-snippet vuln-line). That gives 32 vulnerabilities with exact file and line, chosen by people who never saw my skill.

A finding counts only if it cites the marked file and line. Same prompt, quick mode, static analysis only:

Agent Found with file and line Within 15 lines Findings
Claude (Opus, two cloud runs) 23/32 and 19/32 27 and 23 37 and 36
Codex 18/32 22/32 28
Antigravity (Gemini) 6/32 9/32 12

Two things surprised me.

The model matters, but so does the plumbing. Earlier standard runs on my PC (older Claude model) scored 7/32, 7/32 and 8/32. Reading the transcripts, the agent had seen the client-side XSS, the SSRF, the chatbot and the password-reset issues. It wrote them in the checklist as "noted" or "known surface" and never promoted them to findings. Adding a word filter did not help: the agent used a synonym. What worked was structural: the deterministic pre-pass now writes one row per dangerous-code category (HTML without escaping, code execution, deserialization, SSRF, client routes, seed credentials…), and the delivery check fails unless each row is closed with a finding or a "not a defect" that cites a file and line it read. The next two runs: 13/32 and 12/32, same model.

Antigravity writes few findings. It found real things, but stopped early. I have not solved this yet; it is on the roadmap.

The misses are shared. All 9 vulnerabilities Claude missed in its best run were missed by Codex and Antigravity too, and they fall into three groups:

Pattern-like bugs (injection, XSS, authentication) get caught. Flaws that need intent or outside knowledge don't. That is the next thing I want to measure.

These are mine, so read them with the caveat above. Each has planted defects plus decoys, things that look wrong but are fine (a session cookie, data published because the law requires it, an IP log deleted after 7 days). A false alarm on a decoy counts against the skill.

Claude Codex Antigravity
appsec, Django (11 defects, 5 decoys) 11/11, 0 false alarms 11/11, 0 10/11, 0
gdpr, school website in PHP (15 defects, 5 decoys) 15/15, 0 14/15, 0 14/15, 0

Codex can start a subagent; many setups cannot. For those, refutation_pack.py prepara writes a folder with only the method and one brief per finding: the claim, the code excerpt and the typical ways it could be wrong. It contains no author reasoning. You hand it to a new session (another conversation, another agent, a colleague) and import the answers with refutation_pack.py importa. In a test with four planted false GDPR gaps, the verifier rejected three and downgraded the fourth, and rejected no true gap.

On a real 1,000-file application, two GDPR runs agree on only about 60–69% of the individual gaps: they describe the same problem with different words and granularity. Fixed questions help a lot where they apply. Three fixed matrices (per data entity, per privacy-notice item, and who can read what for each role) bring agreement on those rows to 90–93%, 83–88% and 86–92%. The free-form part is still noisy. If you have ideas, I would love to hear them.

It started as a long audit prompt I used by hand. I turned it into skills with Claude Code: Claude split the prompt into a SKILL.md plus reference files, wrote the Python scripts and the test apps, and ran the benchmarks in subagents. After each run I asked it to read the transcript and find where the agent had cut corners, then we changed the scripts and measured again. Most of the rules above exist because a run broke them.

It does not certify anything, it does not replace a pentest or a DPO, and a static read is not proof that the running site behaves the same way. The benchmarks are small. The skill's own instructions are in Italian (the agent answers in your language).

/plugin marketplace add dextsamu/appsec-sentinel
/plugin install appsec-sentinel@appsec-sentinel
/plugin install gdpr-sentinel@appsec-sentinel

Codex and Antigravity use the same SKILL.md convention (install paths in the repo docs). Then ask, for example: "Use appsec-sentinel. Quick mode, static analysis only, on this folder. The repo is mine; no dynamic testing is authorized." Commands work in English and Italian.

tests/. You can rerun everything. Feedback, criticism and new benchmarks are very welcome, and a star helps other people find the project.

── more in #ai-agents 4 stories · sorted by recency
── more on @claude 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/i-benchmarked-a-secu…] indexed:0 read:5min 2026-11-16 · —