A coding agent can find real vulnerabilities, but only as one layer, and only with a check behind each finding. This playbook sets up three security passes for one repository: a secret scan and agent self-check before each commit, a rules scanner plus an agent review on every pull request, and a weekly full-repository agent scan with dependency monitoring. By my estimate the setup takes an afternoon. Commands and versions were checked against each tool's docs on 23 October 2026.
Can a coding agent find vulnerabilities in your codebase? #
Yes: agents helped find the two remote code execution bugs in PHP itself that were fixed on 21 October 2026. Calif's Ryan Austin found the first on 1 August under OpenAI's Patch the Planet program, using GPT-5.6 Sol, and Wiz found both independently from 19 August. PHP shipped fixes for CVE-2026-109618 and CVE-2026-109619, both rated CVSS v4.0 9.2 (pre-announcement), in 8.5.12, 8.4.27, 8.3.36 and 8.2.35 on 21 October (php.net).
Each finding came with a check. PHP confirmed Calif's proof of concept on 15 August, by Calif's timeline, and Wiz's single script gave unauthenticated code execution on the stock WordPress, Drupal and Joomla containers (Wiz). Wiz's headline figure, "affecting 70% of the Internet", is PHP's market share as Wiz states it, without a source, not a count of vulnerable servers.
The NIST curve result of 16 October has the same shape. Anthropic says an internal model worked out "in under half an hour" that the P-curve seeds came from the POSIX rand48 generator seeded with 1997-98 timestamps (Anthropic). Each recovered state regenerates all 160 bits of its seed, and Filippo Valsorda reimplemented it in Go to verify the claim. The $36,864 bounty goes to ISRG. It is "not a cryptographic break", and says nothing about your repository.
The measurement that does is older. Semgrep pointed Claude Code and Codex at 11 Python web apps in 2025 and triaged 445 findings by hand: 86% of Claude Code's were false positives and 82% of Codex's. Of the IDORs Claude Code reported, 22% were real; of its SQL injections, 5%. Three identical runs returned 3, 6 and 11 findings (Semgrep). Those were Sonnet 4 and o4-mini, and I found no 2026 rerun.
What you need #
- A Git repository on GitHub with an
originremote, and permission to add workflows and secrets - pre-commit 4.x andGitleaks (v8.30.1 is current)
- Claude Code with a paid plan or API key, or OpenAI Codex Security access (CLI 0.6.0 on npm, Node 22.13+, 24 or 26, and Python 3.10+)
- Semgrep (free up to 10 contributors) orCodeQL (free on public repositories)
- For dependency monitoring, Syft and aDependency-Track server on Docker
1. Block secrets and review the diff before each commit #
The pre-commit layer catches leaked keys and obvious mistakes while they are still on your machine. Gitleaks runs as a pre-commit hook on staged files. Its README pins an older rev, so run pre-commit autoupdate after adding it, then pre-commit install.
repos:
- repo: https://github.com/gitleaks/gitleaks
rev: v8.24.2
hooks:
- id: gitleaks
A staged key stops the commit:
Detect hardcoded secrets.................................................Failed
For the agent itself, Claude Code's security guidance plugin adds a pattern check after each edit and a diff review at the end of each turn. Before pushing, /security-review reviews "the diff between your branch and origin's default branch."
/plugin install security-guidance@claude-plugins-official
My own pipeline sits at this layer. make security runs govulncheck, pnpm audit --prod, gosec on changed lines and Gitleaks on the diff before every task commit, and takes under a minute. It does not run in CI yet.
2. Gate pull requests with a rules scanner and an agent review #
The pull request layer blocks a merge, so it has to be deterministic first and agent second. Rules scanners follow taint across files, which Semgrep found agents do worst. CodeQL default setup needs no YAML: Settings, Advanced Security, then Set up, Default and Enable CodeQL next to CodeQL analysis (GitHub docs). Semgrep's sample config runs in its own container:
container:
image: semgrep/semgrep
if: (github.actor != 'dependabot[bot]')
steps:
- uses: actions/checkout@v6
- run: semgrep ci
env:
SEMGREP_APP_TOKEN: ${{ secrets.SEMGREP_APP_TOKEN }}
Without an account, the step is semgrep scan --config auto. For the agent review, Anthropic's security review Action comments on the lines it flags. The README's full workflow adds a checkout step and the pull-requests: write permission around this step:
- uses: anthropics/claude-code-security-review@main
with:
comment-pr: true
claude-api-key: ${{ secrets.CLAUDE_API_KEY }}
Set claude-model explicitly: the README still defaults to Claude Opus 4.1. The Action filters out denial-of-service and "Memory/CPU exhaustion issues" by default. CVE-2026-109618 is an out-of-memory handling bug, so a project where availability matters should edit that filter. The README also warns it "is not hardened against prompt injection attacks and should only be used to review trusted PRs."
On OpenAI's side, the Codex Security GitHub Action runs with scope: diff. Start it with fail-on-severity: none and raise the threshold once you trust its findings.
3. Schedule a full-repository scan and dependency monitoring #
A full-repository agent scan finds what no single diff shows, at a cost per run, so it belongs on a weekly schedule and before releases. The Codex Security CLI writes a report, findings and a coverage file outside the repository (CLI docs):
npx @openai/codex-security login
npx @openai/codex-security scan "$REPOSITORY" \
--auth chatgpt --output-dir "$SCAN_DIR"
FINDINGS 2 (2 confirmed this scan; 0 previously found; 1 high, 1 medium)
ELAPSED 42s
The same Action runs it weekly with scope: repository. Add max-cost, an estimated cost threshold that the docs say is "not a hard spending cap". In Claude Code, /plugin install claude-security@claude-plugins-official and then /claude-security gives a multi-agent scan with SARIF output and patches it never applies by itself (docs).
Dependency scanners only know a vulnerability once it has an advisory, and monitoring catches that advisory the day it appears in a component you ship. Generate an SBOM with syft <image> -o cyclonedx-json, scanning the container image rather than only the lockfile so the runtime is included, and upload it to Dependency-Track, which re-analyses every component daily.
The PHP leak showed why this layer has to move fast. AWS saw an exploitation attempt on 17 October, four days before disclosure, and advises investigating from that date rather than from the CVE (AWS).
Check that each layer catches something #
Plant a fake key on a throwaway branch and expect the Gitleaks hook to fail. Plant a known IDOR and a known SQL injection, and confirm the pull request gate flags both. In Semgrep's 2025 study the agents' SQL injection reports were the least reliable, which is why a rules scanner sits in the same gate. For Codex Security, read scan.status in scan-manifest.json and coverage.json, because "exit 0 doesn't mean every file was reviewed."
What goes wrong #
False positives flood the pull request
Semgrep's 82-86% false-positive rates, and the note in OpenAI's Patch the Planet post that frontier models "produce a high volume of false positives", mean an agent gate that blocks merges on day one gets switched off by day three. Keep it advisory, ask for validation evidence, and pass project-specific false-positive-filtering-instructions to the Claude Action.
One clean run proves nothing
Agent scans are nondeterministic: Claude's docs say "two scans of the same code can surface different findings." Run the full scan on a schedule, and use the revision stamp Claude Security writes to tie each report to a commit.
Skipped jobs look like passed jobs
git commit --no-verify skips the hook, and the Codex example skips fork and Dependabot pull requests: "A skipped job is not evidence of a security review." GitHub counts a skipped job as passed even for a required check, so the Codex docs suggest a separate gate job that requires a completed scan. Treat the hook as convenience.
A proof of concept that cheats
Elias Taylor found that public PHP2Shell exploits relied on a file inclusion the lab setup added, not on stock PHP. Rerun any agent-written exploit against an unmodified target.
Checklist #
- Gitleaks pre-commit hook installed and autoupdated
/security-reviewor the security guidance plugin before each push- CodeQL default setup or
semgrep cirequired on pull requests - Agent review on pull requests, advisory, with the model set explicitly
- Weekly full-repository agent scan with a cost limit
- SBOM of the deployed image in Dependency-Track
- Seeded-bug test passed by every layer
This is the move pull request code review with a security pass in front of the human. The verification layers post covers the checks around it, and the DevOps tools developers list in their stacks include the scanners.
New posts by email #
About one email a week with what we published. No account needed.
Which of these do you use? #
Nobody on Stackness runs GPT yet. Tick what you use and be the first.