During my internship at RakFort, one of my tasks was simple to describe and slow to do: run every security check in Secfoo with the Claude agent, and then check whether the AI was actually right.
Not "did the run finish". Not "does the report look good". Is each finding true?
This post is about what I found, and what I'd tell anyone who is starting to use AI for security reviews.
Secfoo is an open-source tool that turns the AI coding agent you already use (Claude Code, Codex, Gemini CLI, Cursor, GitHub Copilot and others) into a security reviewer.
You point it at your code, pick a check, and it gives the AI clear instructions and a fixed report format. You get back a structured report with findings, severity and recommendations, and everything shows up in a local dashboard.
secfoo run --skill deployment-readiness --agent claude --target .
secfoo serve # open the dashboard
It has 9 checks, including SAST, threat modeling, secret scanning, dependency triage, deployment readiness and responsible AI compliance.
My target was Secfoo's own codebase (a good test, because I could read every file the AI talked about).
For each report I:
Step 3 matters more than I expected. Code changes every day. If you check an old report against today's code, you'll blame the AI for things that were true at the time.
| # | Severity | Finding | Verdict |
|---|---|---|---|
| F1 | High | Local dashboard has no login or CSRF protection | ✅ Confirmed (exact lines were right) |
| F2 | High | No security scanning in CI | ⚠️ Partly wrong |
| F3 | Medium | Release actions pinned by tag, long-lived tokens | ✅ Confirmed |
| F4 | Medium | Uploads saved to disk before the size check | ✅ Confirmed |
| F5 | Medium | Docker image runs as root, base image not pinned | ✅ Confirmed |
| F6 | Low | No health endpoint | ✅ Confirmed |
| F7 | Low | Dependencies have no upper version pins | ✅ Confirmed |
All four findings were accurate, and the overall rating ("Low") made sense for a tool that doesn't make decisions about people.
Most of F2 was right. CI really didn't run any security scanners, and the linter was set to never fail the build.
But the report also said the release workflow "runs on any v tag push"*.
It didn't. The workflow had been changed to manual-only five days before the scan. The file even had a comment at the top explaining the change.
So the AI described how the project used to work. My guess is that it matched a very common pattern ("release on tag push") instead of reading the current trigger carefully.
This is exactly the kind of mistake that is easy to miss:
If I had just skimmed the report, I would have believed it.
Bad input. I fed the CLI things a real user might get wrong: a folder that doesn't exist, a file instead of a folder, an unknown skill or agent name, a GitHub repo that doesn't exist. Every one gave a clear error, with no crash and no AI tokens wasted. I found two small issues (a negative --timeout was accepted, and an empty target quietly scanned the current folder) and reported them.
The dashboard. All pages loaded and the finding counts matched the reports, except the home page, which crashed with a 500 error. The cause was a recent change: agents that don't report cost now store "unknown" instead of $0, and the home page added the costs together with a plain sum(), which fails on an empty value. It was a one-line fix, and my PR for it (#31) is now merged.
1. AI reviewers are genuinely useful. Most findings were correct, with the right file and line numbers. For coverage and speed, it's a huge help, especially for a small team.
2. "Looks professional" is not the same as "true". The one wrong detail was written just as confidently as the correct ones. Severity labels and neat tables don't make a finding correct.
3. Check against the right version of the code. Use the commit from the scan date. Otherwise you'll mark correct findings as wrong (or the other way round).
4. Keep a human in the loop for anything that matters. If a report is going to someone who will sign it off, every finding should be checked first. It took me a while, but it's much faster than reviewing the code from scratch.
Secfoo is free and open source (MIT). If you're a student or developer interested in security, try it on your own project and check the findings yourself. That's honestly the best way to learn what AI reviewers are good and bad at.
👉 https://github.com/secfoo-com/secfoo
Feedback, issues and contributions are welcome. I'd love to hear how it does on your code!
This is my second post about Secfoo. The first one was about catching an AI scanner inventing fake vulnerabilities.