{"slug": "i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up", "title": "I Fact-Checked Every Finding an AI Security Reviewer Gave Me. Here's What Held Up.", "summary": "An intern at RakFort fact-checked every finding produced by Secfoo, an open-source tool that turns AI coding agents like Claude Code and Codex into security reviewers, against the tool's own codebase. Six of seven findings held up with correct file and line references, but one claim that the release workflow runs on any v* tag push was wrong—the workflow had been made manual-only five days earlier. The intern also found a dashboard 500 error caused by summing an \"unknown\" cost value, and submitted a one-line fix that was merged.", "body_md": "During my internship at RakFort, one of my tasks was simple to describe and slow to do: **run every security check in [Secfoo](https://github.com/secfoo-com/secfoo) with the Claude agent, and then check whether the AI was actually right.**\n\nNot \"did the run finish\". Not \"does the report look good\". Is each finding *true*?\n\nThis post is about what I found, and what I'd tell anyone who is starting to use AI for security reviews.\n\nSecfoo is an open-source tool that turns the AI coding agent you already use (Claude Code, Codex, Gemini CLI, Cursor, GitHub Copilot and others) into a security reviewer.\n\nYou point it at your code, pick a check, and it gives the AI clear instructions and a fixed report format. You get back a structured report with findings, severity and recommendations, and everything shows up in a local dashboard.\n\n```\nsecfoo run --skill deployment-readiness --agent claude --target .\nsecfoo serve   # open the dashboard\n```\n\nIt has 9 checks, including SAST, threat modeling, secret scanning, dependency triage, deployment readiness and responsible AI compliance.\n\nMy target was Secfoo's own codebase (a good test, because I could read every file the AI talked about).\n\nFor each report I:\n\nStep 3 matters more than I expected. Code changes every day. If you check an old report against today's code, you'll blame the AI for things that were true at the time.\n\n| # | Severity | Finding | Verdict | \n|---|---|---|---|\n| F1 | High | Local dashboard has no login or CSRF protection | ✅ Confirmed (exact lines were right) | \n| F2 | High | No security scanning in CI | ⚠️ Partly wrong | \n| F3 | Medium | Release actions pinned by tag, long-lived tokens | ✅ Confirmed | \n| F4 | Medium | Uploads saved to disk before the size check | ✅ Confirmed | \n| F5 | Medium | Docker image runs as root, base image not pinned | ✅ Confirmed | \n| F6 | Low | No health endpoint | ✅ Confirmed | \n| F7 | Low | Dependencies have no upper version pins | ✅ Confirmed | \n\nAll four findings were accurate, and the overall rating (\"Low\") made sense for a tool that doesn't make decisions about people.\n\nMost of F2 was right. CI really didn't run any security scanners, and the linter was set to never fail the build.\n\nBut the report also said the release workflow **\"runs on any v* tag push\"**.\n\nIt didn't. The workflow had been changed to **manual-only** five days before the scan. The file even had a comment at the top explaining the change.\n\nSo the AI described how the project *used to* work. My guess is that it matched a very common pattern (\"release on tag push\") instead of reading the current trigger carefully.\n\nThis is exactly the kind of mistake that is easy to miss:\n\nIf I had just skimmed the report, I would have believed it.\n\n**Bad input.** I fed the CLI things a real user might get wrong: a folder that doesn't exist, a file instead of a folder, an unknown skill or agent name, a GitHub repo that doesn't exist. Every one gave a clear error, with no crash and no AI tokens wasted. I found two small issues (a negative `--timeout` was accepted, and an empty target quietly scanned the current folder) and reported them.\n\n**The dashboard.** All pages loaded and the finding counts matched the reports, except the home page, which crashed with a 500 error. The cause was a recent change: agents that don't report cost now store \"unknown\" instead of $0, and the home page added the costs together with a plain `sum()`, which fails on an empty value. It was a one-line fix, and my PR for it ([#31](https://github.com/secfoo-com/secfoo/pull/31)) is now merged.\n\n**1. AI reviewers are genuinely useful.** Most findings were correct, with the right file and line numbers. For coverage and speed, it's a huge help, especially for a small team.\n\n**2. \"Looks professional\" is not the same as \"true\".** The one wrong detail was written just as confidently as the correct ones. Severity labels and neat tables don't make a finding correct.\n\n**3. Check against the right version of the code.** Use the commit from the scan date. Otherwise you'll mark correct findings as wrong (or the other way round).\n\n**4. Keep a human in the loop for anything that matters.** If a report is going to someone who will sign it off, every finding should be checked first. It took me a while, but it's much faster than reviewing the code from scratch.\n\nSecfoo is free and open source (MIT). If you're a student or developer interested in security, try it on your own project and check the findings yourself. That's honestly the best way to learn what AI reviewers are good and bad at.\n\n👉 [https://github.com/secfoo-com/secfoo](https://github.com/secfoo-com/secfoo)\n\nFeedback, issues and contributions are welcome. I'd love to hear how it does on your code!\n\n*This is my second post about Secfoo. The first one was about [catching an AI scanner inventing fake vulnerabilities](https://dev.to/amir09/secfoo-i-caught-our-security-scanner-inventing-fake-vulnerabilities-heres-the-proof-1hak).*", "url": "https://wpnews.pro/news/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up", "canonical_source": "https://dev.to/amir09/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-heres-what-held-up-e2k", "published_at": "2026-11-01 20:40:57+00:00", "updated_at": "2026-11-01 20:51:58.798415+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools", "artificial-intelligence"], "entities": ["Secfoo", "RakFort", "Claude Code", "Codex", "Gemini CLI", "Cursor", "GitHub Copilot"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up", "markdown": "https://wpnews.pro/news/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up.md", "text": "https://wpnews.pro/news/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up.txt", "jsonld": "https://wpnews.pro/news/i-fact-checked-every-finding-an-ai-security-reviewer-gave-me-here-s-what-held-up.jsonld"}}