{"slug": "don-t-let-the-ai-find-your-bugs-let-it-judge-them", "title": "Don't Let the AI Find Your Bugs. Let It Judge Them.", "summary": "A developer building an LLM-based vulnerability scanner explains why he deliberately designed it to flag safe code, arguing that deterministic taint tracking with false positives is more trustworthy than AI agents that skim codebases and hallucinate vulnerabilities. He rejects the common approach of pointing an AI agent at a codebase because LLMs produce inconsistent results, are poor at exhaustive search, and invent findings.", "body_md": "My vulnerability scanner flagged this Java method as SQL injection:\n\n```\nString id = request.getParameter(\"id\");        // user input\nif (!id.matches(\"[0-9]+\")) {                   // digits only, or throw\n    throw new IllegalArgumentException(\"id must be numeric\");\n}\nString sql = \"DELETE FROM products WHERE id = \" + id;\nstmt.executeUpdate(sql);                       // flagged as SQL injection\n```\n\nLook at line 2. The input must be pure digits or the method throws. You cannot\n\ninject SQL through `[0-9]+`\n\n. There is no attack here. The scanner flagged safe code.\n\nHere's the part that sounds wrong: **I designed it to do that.** The false alarm is\n\nnot a bug in my scanner. It's the plan.\n\nLet me explain, because this decision is the entire foundation of the scanner —\n\nand I think it's the decision most people building \"AI security tools\" right now\n\nare getting backwards.\n\nI'm building an LLM-based vulnerability scanner — in public, like everything I\n\nship. The pitch is simple: AI writes a lot of code now, fast, often for people\n\nwho aren't security experts. That code ships with holes. Someone has to find\n\nthem.\n\nSo the obvious idea — the one I started with — is the one you've seen in a hundred\n\nlaunch posts this year:\n\n\"Point an AI agent at your codebase. It reads everything and finds your\n\nvulnerabilities.\"\n\nI killed that idea before writing a line of code. Not because it doesn't sound\n\namazing. Because I couldn't defend it with numbers — and the reasons it can't be\n\nmeasured are the same reasons you shouldn't trust it in production.\n\nNew to this?Two words carry the whole article. Asourceis where\n\nuntrusted input enters your program (`request.getParameter`\n\n). Asinkis\n\nwhere it becomes dangerous (`executeUpdate`\n\n— running a database command). A\n\nvulnerability is untrusted data reaching a sink without being cleaned on the\n\nway. That's it; everything below builds on those two words.\n\nAsk an LLM to audit a codebase and you hit three walls.\n\n**Wall 1: You get a different answer every run.** LLMs are probabilistic. Same\n\nrepo, same prompt, run it twice — the bug lists don't match. Which run goes in\n\nthe security report? Which one do you benchmark? When a tool's output changes\n\nbetween runs, you can't measure it, you can't compare it to Semgrep or CodeQL,\n\nand you can't do science with it. You can only do demos.\n\n**Wall 2: Searching is the thing LLMs are bad at.** Careful, exhaustive\n\nenumeration over a large space — visit every file, check every call, miss\n\nnothing — is exactly what LLMs don't do. They skim. They fixate. They get bored\n\nin the middle of a long file (position bias is real). A scanner that \"usually\n\nchecks most of the code\" is not a scanner.\n\n**Wall 3: They invent.** In my other project, an AI chatbot invented products\n\nthat didn't exist in the store. An AI bug-hunter does the same thing with\n\nvulnerabilities — confident reports about code that isn't there. Now your\n\nsecurity tool hallucinates CVEs. Great.\n\nIf you've read my earlier article about [the chatbot that lied to customers](https://dev.to/alimafana/your-ai-is-lying-to-your-customers-and-prompt-engineering-wont-fix-it-5408),\n\nyou know where this goes: **you can't prompt your way out of a structural problem.**\n\nThis is the same disease in a different body.\n\nFine — so use deterministic tools. Pattern rules, taint tracking. Semgrep and\n\nCodeQL have done it for years.\n\nBut rules have their own wall, and my flagged-but-safe method above IS that wall.\n\nA taint tracker follows data: user input enters at `getParameter`\n\n(the *source*),\n\ntravels through variables, and reaches `executeUpdate`\n\n(the *sink* — the\n\ndangerous operation). Path exists → alarm. That's the whole trick, and it's a\n\ngood trick. It's deterministic, fast, and it never gets bored.\n\nBut look at the method again. The data DOES flow from source to sink. The taint\n\ntracker is not wrong about the flow. It's wrong about the *meaning* — the\n\n`matches(\"[0-9]+\")`\n\nguard makes the flow harmless, and understanding that\n\nrequires understanding what the code *means*, not just where the data *goes*.\n\nRules can't read meaning. So real-world scanners over-flag, developers drown in\n\nfalse alarms, and — every security team knows this story — they stop reading the\n\nreports. The industry's own benchmark makes the point better than I can: the\n\nOWASP Benchmark (2,740 labeled Java test cases, the standard test set for Java\n\nscanners) contains **701 cases in my scanner's categories that are deliberately\ndesigned to trick tools into false alarms.** Nearly half the test set exists just\n\nSo:\n\nEveryone's strength is the other one's weakness. You can see the answer coming.\n\nPolice detectives don't sentence anyone. Judges don't collect evidence. The\n\nsystem splits the work on purpose: exhaustive, procedural search by one party;\n\ncareful judgment of each individual case by another.\n\nThat's the architecture:\n\nDeterministic rules do ALL the searching. The LLM only judges what the rules\n\nfound. A memory of real-world bugs helps it judge better.\n\nConcretely, the pipeline looks like this:\n\n```\nsource code\n   → Joern builds a Code Property Graph        (code → queryable data-flow graph)\n   → fixed taint queries find candidates       (same input = same output, always)\n   → slicer cuts a ~small snippet per candidate (the evidence file)\n   → RAG fetches similar known CVEs            (past cases, from GitHub advisories)\n   → LLM judges ONE candidate at a time        (real bug? severity? fix?)\n```\n\nThe detective is [Joern](https://joern.io) — an open-source static analysis tool\n\nthat turns code into a graph you can query. My whole detection layer is a small\n\ntable of rules. Adding a vulnerability class is one row:\n\n```\ncase class VulnClass(\n    name: String,                // \"sql-injection\"\n    cwe: String,                 // \"CWE-89\"\n    sinkNames: String,           // which call is dangerous\n    sinkMethodFullName: String,  // narrowed to the right type\n    sinkArg: Int,                // which argument carries the taint (0 = any)\n    sinkReceiver: String = \"\"    // fallback when the type can't be resolved\n)\n\nval classes = List(\n  VulnClass(\"sql-injection\", \"CWE-89\",\n    \"executeQuery|executeUpdate|executeLargeUpdate|execute|addBatch|prepareStatement|prepareCall|nativeSQL\",\n    \".*(java|javax)\\\\.sql\\\\..*\", 1),\n  VulnClass(\"command-injection\", \"CWE-78\", \"exec\",           \".*Runtime.*\",        0),\n  VulnClass(\"command-injection\", \"CWE-78\", \"<init>|command\", \".*ProcessBuilder.*\", 0),\n  VulnClass(\"path-traversal\", \"CWE-22\", \"<init>\",\n    \".*java\\\\.io\\\\.(File|FileInputStream|FileOutputStream|FileReader|FileWriter|RandomAccessFile)\\\\.<init>.*\", 0),\n  // ... 3 more rows (JdbcTemplate SQLi, Paths.get, XSS)\n)\n\n// for each row: is there a data-flow path from source to sink?\nsink.reachableByFlows(source)\n```\n\n(Sources — `getParameter`\n\n, `getHeader`\n\n, `getCookies`\n\nand the rest of the HTTP\n\nrequest surface — are shared by every class, so they live in one regex above the\n\ntable rather than being repeated on each row.)\n\nOne line — `reachableByFlows`\n\n— is the entire detector. It over-flags by design\n\n(it flagged our digits-only method), and that's fine. Detectives are *supposed*\n\nto bring in every plausible suspect. Their job is to miss nothing.\n\nThe judge is Gemma (via Google's API), and this is the important part: **it\nnever sees the codebase.** It sees one small slice of code — the few lines the\n\n```\n{\n  \"reasoning\":   \"≤ 3 sentences, generated FIRST, before the verdict\",\n  \"confirmed\":   true,\n  \"severity\":    \"Critical | High | Medium | Low  (CVSS bands)\",\n  \"explanation\": \"the concrete risk\",\n  \"fix\":         \"corrected code\"\n}\n```\n\nJudging one small snippet is the task LLMs are genuinely good at. There's\n\nnothing to search, nothing to miss, nothing to invent — the evidence is the\n\nwhole context. And because discovery is frozen, the *same* suspects go to the\n\njudge every run. The randomness is contained to the one layer where a second\n\nopinion is the point.\n\n(One prompt detail that gets a full article later in the series: you have to actively stop\n\nthe judge from agreeing with the detective. Tell a model \"a scanner flagged\n\nthis\" and it wants to say yes. The prompt explicitly states that rejecting a\n\nfinding is a correct and expected answer — otherwise the judge just rubber-stamps\n\nevery arrest, and you've built an expensive echo.)\n\nI test on a small OWASP-style sample set: 8 methods — 4 real vulnerabilities\n\nacross SQL injection, command injection, path traversal, and XSS; 2 methods that\n\nare **flagged by the rules but actually sanitized** (the digits-only method\n\nabove is one); and 2 genuinely safe methods.\n\n| Stage | Result |\n|---|---|\n| Discovery (rules) | found all 6 flows: the 4 real + the 2 sanitized traps. Silent on the 2 safe methods. |\n| Judge (an earlier, smaller Gemma) |\n6/6 correct — confirmed the 4 real (with CVSS severities), rejected both sanitized false alarms\n|\n\nThat rejection line is the thesis in one row. The deterministic layer cannot\n\nrecognize a sanitizer. The LLM can. Division of labor works.\n\nAnd the architecture scales further than I expected for something run from a\n\nlaptop: on the full OWASP Benchmark, Joern built the graph over **2,766 files in\n100 seconds** and ran the full query library — all seven rows — in\n\nThen I ran it against the real thing — the full OWASP Benchmark, 1,478 labeled\n\ncases in my four categories, scored against Semgrep and CodeQL by one program\n\nreading everyone's SARIF:\n\n| tool | precision | recall | F1 |\n|---|---|---|---|\n| CodeQL | 0.65 | 1.00 | 0.78 |\n| Semgrep | 0.60 | 0.86 | 0.70 |\n| my rules alone | 0.56 | 1.00 |\n0.72 |\nmy rules + the judge (sample-derived) |\n0.67 |\n0.98 | ~0.79 |\n\nThe judge removed **52% of the false alarms** and cost **2% of the real bugs**\n\nin the latest verified run. One note before you check my arithmetic: the judged\n\nrow is measured *inside* a 200-case stratified sample that deliberately\n\nover-samples false alarms (discovery precision there starts at 0.51, not 0.56)\n\n— so 0.67 is a measured number, not \"614 minus 52%\". That caveat ships with the\n\nnumber.\n\nXSS went from 0.50 to 0.78 precision — those are the `ESAPI.encodeForHTML()`\n\ntraps, exactly the \"is this actually safe?\" question a taint engine can't answer.\n\nMy rules find every one of the 777 real vulnerabilities — the same recall as\n\nCodeQL, from that seven-row table. But read the precision column before you get\n\nexcited: CodeQL reaches perfect recall with 427 false positives, and I need 614\n\nto do it. The difference is years of hand-built sanitizer knowledge that a\n\nseven-row table doesn't have.\n\nHonesty section, because this is an engineering log and not a launch post:\n\nCodeQL beats my rules layer on F1, 0.78 to 0.72, and I'd rather say that than\n\nhunt for a framing where I win. The claim I can defend is narrower and more\n\ninteresting — a seven-row rule table plus a 31B open model lands in the same F1\n\nrange as a mature commercial engine, and the judge is what closes the precision\n\ngap.\n\n**1. Give the probabilistic system the judgment task, never the coverage task.**\n\nAny job where \"misses nothing, same answer every time\" matters — searching,\n\nenumerating, auditing — belongs to deterministic code. The LLM gets the job\n\nwhere meaning matters and the input is small. This split applies way beyond\n\nsecurity: it's the same reason my chatbot searches the product database with SQL\n\nand only lets the LLM *phrase* the answer.\n\n**2. If you can't re-run it, you can't measure it.** Deterministic discovery\n\nmeans every experiment is repeatable: same code in, same candidates out, and any\n\nchange in results traces to the one layer I changed. The moment discovery is\n\nprobabilistic, comparisons against other tools become vibes.\n\n**3. Over-flagging is fine if someone competent reviews the flags.** I stopped\n\ntrying to make the rules smart. Rules that catch everything + a judge that\n\nunderstands meaning beat clever rules with no judge. Design each layer to fail\n\nin the direction the next layer can fix.\n\nThe academic version of this argument exists too — a 2025 paper called LLMxCPG\n\n(arXiv 2507.16585) pairs Code Property Graphs with LLM judgment the same way,\n\nwhich told me the instinct wasn't just mine. The part I hadn't seen done in\n\npublic is the one above: running it against the incumbents on the industry's own\n\nbenchmark and publishing every number, including the losses.\n\nThis is the first article in a series where I do that in the open. Next: the\n\nthree words that explain almost every injection bug — source, sink, and taint —\n\nwritten for anyone who's never done security work. After that, the false-alarm\n\nproblem in detail, and what happened when I gave the same 200 code snippets to\n\nthree different AI judges — including why the frontier model didn't win.\n\n*I'm Ali Afana — AI builder and security researcher, writing from Gaza. I\nbuild systems in public, measure them against ground truth, and keep the\nreceipts. This scanner is one project on a longer road — follow for what\ncomes next.*", "url": "https://wpnews.pro/news/don-t-let-the-ai-find-your-bugs-let-it-judge-them", "canonical_source": "https://dev.to/alimafana/dont-let-the-ai-find-your-bugs-let-it-judge-them-5dbp", "published_at": "2026-08-13 22:44:53+00:00", "updated_at": "2026-08-13 23:16:18.365097+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-safety", "ai-tools", "developer-tools"], "entities": ["Semgrep", "CodeQL"], "alternates": {"html": "https://wpnews.pro/news/don-t-let-the-ai-find-your-bugs-let-it-judge-them", "markdown": "https://wpnews.pro/news/don-t-let-the-ai-find-your-bugs-let-it-judge-them.md", "text": "https://wpnews.pro/news/don-t-let-the-ai-find-your-bugs-let-it-judge-them.txt", "jsonld": "https://wpnews.pro/news/don-t-let-the-ai-find-your-bugs-let-it-judge-them.jsonld"}}