# I benchmarked a security-audit skill on Claude, Codex and Antigravity. Here are the numbers.

> Source: <https://dev.to/dextsamu/i-benchmarked-a-security-audit-skill-on-claude-codex-and-antigravity-here-are-the-numbers-cc6>
> Published: 2026-11-16 09:46:24+00:00

Most "AI security audit" prompts I have seen share the same problem: the report looks great and you cannot tell what was actually checked. An agent that writes "authentication: verified" after reading one file is worse than no audit, because someone will set a budget on it.

I work with Italian public administrations and non-profits, so I built two open-source skills for AI coding agents that try to fix this:

Both are Apache-2.0, work in English and Italian, and run on Claude Code, Codex and Antigravity. This post is about how I measured them, including the results that are not flattering.

`check_delivery.py` fails the audit if a dangerous-code category found by the pre-pass is left without a verdict, if forbidden words like "compliant" appear, or if secret values leak into the folder.
My own test apps have a weakness: I wrote the bugs and I wrote the checklist, so they measure obedience more than generalization. Juice Shop is better: its authors mark the vulnerable lines of their coding challenges in the source (`vuln-code-snippet vuln-line`). That gives **32 vulnerabilities with exact file and line**, chosen by people who never saw my skill.

A finding counts only if it cites the marked file and line. Same prompt, `quick` mode, static analysis only:

| Agent | Found with file and line | Within 15 lines | Findings | 
|---|---|---|---|
| Claude (Opus, two cloud runs) | **23/32** and 19/32 | 27 and 23 | 37 and 36 | 
| Codex | 18/32 | 22/32 | 28 | 
| Antigravity (Gemini) | 6/32 | 9/32 | 12 | 

Two things surprised me.

**The model matters, but so does the plumbing.** Earlier `standard` runs on my PC (older Claude model) scored **7/32, 7/32 and 8/32**. Reading the transcripts, the agent *had seen* the client-side XSS, the SSRF, the chatbot and the password-reset issues. It wrote them in the checklist as "noted" or "known surface" and never promoted them to findings. Adding a word filter did not help: the agent used a synonym. What worked was structural: the deterministic pre-pass now writes one row per dangerous-code category (HTML without escaping, code execution, deserialization, SSRF, client routes, seed credentials…), and the delivery check fails unless each row is closed with a finding or a "not a defect" that cites a file and line it read. The next two runs: **13/32 and 12/32**, same model.

**Antigravity writes few findings.** It found real things, but stopped early. I have not solved this yet; it is on the roadmap.

**The misses are shared.** All 9 vulnerabilities Claude missed in its best run were missed by Codex and Antigravity too, and they fall into three groups:

Pattern-like bugs (injection, XSS, authentication) get caught. Flaws that need intent or outside knowledge don't. That is the next thing I want to measure.

These are mine, so read them with the caveat above. Each has planted defects plus **decoys**, things that look wrong but are fine (a session cookie, data published because the law requires it, an IP log deleted after 7 days). A false alarm on a decoy counts against the skill.

|  | Claude | Codex | Antigravity | 
|---|---|---|---|
| appsec, Django (11 defects, 5 decoys) | 11/11, 0 false alarms | 11/11, 0 | 10/11, 0 | 
| gdpr, school website in PHP (15 defects, 5 decoys) | 15/15, 0 | 14/15, 0 | 14/15, 0 | 

Codex can start a subagent; many setups cannot. For those, `refutation_pack.py prepara` writes a folder with only the method and one brief per finding: the claim, the code excerpt and the typical ways it could be wrong. It contains no author reasoning. You hand it to a *new* session (another conversation, another agent, a colleague) and import the answers with `refutation_pack.py importa`. In a test with four planted false GDPR gaps, the verifier rejected three and downgraded the fourth, and rejected no true gap.

On a real 1,000-file application, two GDPR runs agree on only about 60–69% of the individual gaps: they describe the same problem with different words and granularity. Fixed questions help a lot where they apply. Three fixed matrices (per data entity, per privacy-notice item, and who can read what for each role) bring agreement on those rows to 90–93%, 83–88% and 86–92%. The free-form part is still noisy. If you have ideas, I would love to hear them.

It started as a long audit prompt I used by hand. I turned it into skills with Claude Code: Claude split the prompt into a `SKILL.md` plus reference files, wrote the Python scripts and the test apps, and ran the benchmarks in subagents. After each run I asked it to read the transcript and find where the agent had cut corners, then we changed the scripts and measured again. Most of the rules above exist because a run broke them.

It does not certify anything, it does not replace a pentest or a DPO, and a static read is not proof that the running site behaves the same way. The benchmarks are small. The skill's own instructions are in Italian (the agent answers in your language).

```
# Claude Code
/plugin marketplace add dextsamu/appsec-sentinel
/plugin install appsec-sentinel@appsec-sentinel
/plugin install gdpr-sentinel@appsec-sentinel
```

Codex and Antigravity use the same `SKILL.md` convention (install paths in the repo docs). Then ask, for example: *"Use appsec-sentinel. Quick mode, static analysis only, on this folder. The repo is mine; no dynamic testing is authorized."* Commands work in English and Italian.

`tests/`. You can rerun everything.
Feedback, criticism and new benchmarks are very welcome, and a star helps other people find the project.
