{"slug": "let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug", "title": "Let an AI clear out your false positives without letting it hide a real bug", "summary": "Synapse, a security gate tool, introduces AI-powered false-positive triage that never deletes findings. The system uses a two-model consensus: a proposer model suggests a false-positive verdict, and a separate verifier model must independently agree. Refuted findings are retained and marked but exempt from the gate's exit code, preventing AI from hiding real bugs.", "body_md": "Last time I wrote about wiring up a security gate that blocks merges in CI with [Synapse](https://github.com/KKloudTarus/synapse-ce). Someone left a comment that hit the exact sore spot:\n\n\"The gate is the easy part. Two weeks in, the team turns it off because it's red over false positives every single time.\"\n\nYep. A gate that cries wolf too often dies on its own. People start hitting \"merge anyway,\" and by the time a real vulnerability shows up, nobody's looking anymore.\n\nThe obvious 2020s fix is to hand the whole pile to an LLM and say \"delete the junk.\" Except: the moment a security tool lets an AI *delete* findings, you've just built a brand new vulnerability. The model guesses wrong once, a real SQL injection quietly drops off the report, and nobody's the wiser.\n\nSo Synapse does it differently. The AI is allowed to *propose* that a finding is a false positive. It is never allowed to confirm that on its own, and it is never allowed to delete anything. This whole post is a real run. Terminal output at the bottom, nothing faked.\n\nThree rules, and all three exist so you can trust the output:\n\n**1. The model only proposes.** It doesn't return prose, and it doesn't return a delete command. It returns exactly one typed verdict: `verdict`\n\nis one of `refuted | sound | uncertain`\n\n, `driver`\n\nis a closed snake_case token (not a sentence), and `confidence`\n\nis 0–100. That grammar is validated in Go. The model can hallucinate a whole paragraph of reasoning and none of it reaches the report.\n\n**2. A second, different model has to agree.** When the first model proposes \"refuted\" with confidence ≥ 75, a separate verifier model gets called to assess the same finding independently. The refutation only stands if the verifier also says \"refuted\" at ≥ 75. And the verifier is prompted adversarially. Its job is to\n\n**3. Nothing gets deleted.** A \"refuted\" finding is retain-and-mark: it stays in the report, it stays evidence-sealed, it still shows up in the compliance table. It's just exempt from the gate's exit code. The worst a wrong verdict can do is let one finding skip `--fail-on`\n\n. It can never make a finding disappear.\n\nOne more thing, because it matters for the cost: this is the *second* layer. Before any model runs, a deterministic scope classifier already strips the obvious noise: test files, fixtures, that sort of thing. The model only gets the harder calls: is this sink actually attacker-controlled? Is that interpolated value a constant or user input? Model calls cost tokens, so they only run on production-scope, first-party source findings.\n\nIt's opt-in. Four environment variables:\n\n```\nexport SYNAPSE_FP_TRIAGE_ENABLED=true                  # turn on AI false-positive triage\nexport SYNAPSE_LLM_BASE_URL=http://localhost:20128/v1  # any OpenAI-compatible endpoint\nexport SYNAPSE_FP_TRIAGE_MODEL=<proposer-model>        # a cheap/fast model to propose\nexport SYNAPSE_VERIFIER_MODEL=<different-model>         # a DIFFERENT model to verify (two-model consensus)\n```\n\nA few notes:\n\n`SYNAPSE_LLM_BASE_URL`\n\nis a plain OpenAI-style endpoint (`/v1/chat/completions`\n\n). Run it locally with Ollama or vLLM, or point it at OpenAI. Doesn't matter. Add `SYNAPSE_LLM_API_KEY`\n\nif yours needs a key.`SYNAPSE_VERIFIER_MODEL`\n\n(or you set it to the same model), you get single-model mode. It still works, but the real gate is the second model being Then you run the exact same scan command as always:\n\n```\nsynapse-cli scan . --fail-on high\n```\n\nI built a small service with **both real bugs and deliberate false positives**:\n\n`payments/store.py`\n\n: a SQL query built with `db.session.execute(f\"... {account_id}\")`\n\nwhere `account_id`\n\ncomes straight from `request.args`\n\n. That's a `hashlib.md5(...)`\n\ncall used as a `payments/settings.py`\n\n: a hardcoded AWS access key. `reports/ledger.py`\n\n: another f-string SQL query, but the interpolated value is `PERIOD`\n\n, a \n\n``` bash\n$ synapse-cli scan . --fail-on high\n\nSynapse SCA dogfood – payments-service\n  vulnerabilities: 0\n  findings (promoted): 7\n    critical  risk  0.00  Hardcoded AWS access key id (payments/settings.py:4)\n    high      risk  0.00  SQL execution uses dynamic string construction (payments/store.py:14)\n    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (payments/store.py:14)\n    medium    risk  0.00  Weak hash: MD5 (payments/store.py:21)\n    high      risk  0.00  SQL execution uses dynamic string construction (reports/ledger.py:13)\n    high      risk  0.00  Python SQLAlchemy/raw SQL uses dynamic string construction (reports/ledger.py:13)\n    high      risk  0.00  AWS access key ID (payments/settings.py:4)\n\nsynapse-cli: 6 finding(s) at or above high\n$ echo $?\n1\n```\n\nGate is red over **6 findings at or above high**. Two of them (the SQL in `reports/ledger.py`\n\n) are false alarms you'd have to triage by hand.\n\nThe proposer and the verifier are **two different models** on my endpoint:\n\n``` bash\n$ export SYNAPSE_FP_TRIAGE_ENABLED=true\n$ export SYNAPSE_FP_TRIAGE_MODEL=gpt-5.4-mini      # proposer\n$ export SYNAPSE_VERIFIER_MODEL=gpt-5.4            # verifier (a DIFFERENT model)\n$ synapse-cli scan . --fail-on high\n\nsynapse-cli: AI false-positive triage (gpt-5.4-mini, verified by gpt-5.4): critiqued 7 finding(s), 3 suspected false positive(s) held back from the gate\n\nSynapse SCA dogfood – payments-service\n  findings (promoted): 7            # <- still 7, nothing deleted\n  ...\n  compliance: Synapse AppSec Baseline v1.0 – 3/7 controls passing\n    [FAIL] SAB-INJ-1  No injection weaknesses ...\n           - SQL ... (payments/store.py:14)\n           - SQL ... (reports/ledger.py:13)     # <- the FP still shows in compliance\n    ...\n\nsynapse-cli: 2 suspected false positive(s) at or above high held back from the gate by AI triage (reported with a verdict; not deleted)\nsynapse-cli: 4 finding(s) at or above high\n$ echo $?\n1\n```\n\nRead those lines closely:\n\n`critiqued 7 finding(s), 3 suspected false positive(s) held back`\n\n: the AI looked at 7 findings and proposed 3 as false.`2 ... at or above high held back ... (reported with a verdict; not deleted)`\n\n: two of those were high-severity, held back from the gate but `4 finding(s) at or above high`\n\n: the 4 real bugs still count. Gate is still `exit 1`\n\n). The AI couldn't bury a real bug.`findings (promoted): 7`\n\nis unchanged, and the false positives are still sitting in the compliance table. Nothing left the report.This is my favorite part. With `--json`\n\n, every finding carries the AI's verdict next to it:\n\n```\n[critical] Hardcoded AWS access key id (settings.py:4)\n    verdict=sound    confidence=98   suspected_fp=false   driver=hardcoded_secret\n[high]     SQL execution dynamic string (store.py:14)\n    verdict=sound    confidence=99   suspected_fp=false   driver=confirmed_by_review\n[high]     SQLAlchemy raw SQL dynamic (store.py:14)\n    verdict=sound    confidence=99   suspected_fp=false   driver=dynamic_string_sql\n[high]     AWS access key ID (settings.py:4)\n    verdict=sound    confidence=99   suspected_fp=false   driver=literal\n[medium]   Weak hash: MD5 (store.py:21)\n    verdict=refuted  confidence=96   suspected_fp=true    driver=unspecified_refutation\n[high]     SQL execution dynamic string (ledger.py:13)\n    verdict=refuted  confidence=99   suspected_fp=true    driver=constant_literal\n[high]     SQLAlchemy raw SQL dynamic (ledger.py:13)\n    verdict=refuted  confidence=98   suspected_fp=true    driver=constant_literal\n```\n\nLook at the `driver`\n\ncolumn. For `reports/ledger.py`\n\nthe model worked out on its own that the interpolated value is a `constant_literal`\n\n, a constant, not input. For `store.py`\n\nit's `sound`\n\n: real bug, left alone. And `suspected_fp=true`\n\nis the only thing that exempts a finding from the gate. The key detail: because I configured a *different* verifier (the log literally says `verified by gpt-5.4`\n\n), a finding only gets `suspected_fp=true`\n\nwhen **both** models refute it at ≥ 75. Drop the verifier and these exact three refutations still have to clear a second model to stand.\n\nEnd result: you're left with 4 real problems to fix instead of 6-with-2-fake mixed in. And more importantly, not one real bug got dimmed.\n\nRight question. The whole design is built around it:\n\n`driver`\n\nis forced into a closed token grammar, `verdict`\n\nhas to be from a closed vocabulary, anything else is rejected. No LLM gets prose into the report path.Put another way: the worst a bad verdict can do is exempt one finding from the exit code while it stays visible in the report. Not hide a bug.\n\nNothing about the workflow from the last post changes. Add four environment variables to the scan step and you're done:\n\n```\n      - name: Synapse scan (SARIF + gate on high)\n        env:\n          SYNAPSE_FP_TRIAGE_ENABLED: \"true\"\n          SYNAPSE_LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}\n          SYNAPSE_LLM_API_KEY: ${{ secrets.LLM_API_KEY }}\n          SYNAPSE_FP_TRIAGE_MODEL: \"your-proposer-model\"\n          SYNAPSE_VERIFIER_MODEL: \"your-verifier-model\"   # different from the proposer\n        run: synapse-cli scan . --sarif --fail-on high > synapse.sarif\n```\n\nThe gate stays as deterministic as before. The pile of false alarms just stops eroding the team's trust in it. And since nothing gets deleted, the SARIF you push to Code scanning is still complete; the false positives are only marked, not dropped.\n\nOne honest caveat: this is a noise filter, not a replacement for a human. It gets you looking at the 4 real problems instead of 6 muddled ones. You still have to fix those 4.\n\nSynapse is open source (Apache-2.0): ** github.com/KKloudTarus/synapse-ce**.\n\n`driver`\n\nvocabulary, the consensus threshold: it all lives in `internal/usecase/fptriage`\n\n, and it's short and easy to read.If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.", "url": "https://wpnews.pro/news/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug", "canonical_source": "https://dev.to/aws-builders/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug-1akl", "published_at": "2026-07-12 11:07:40+00:00", "updated_at": "2026-07-12 11:43:36.657260+00:00", "lang": "en", "topics": ["ai-safety", "ai-agents", "developer-tools", "artificial-intelligence", "large-language-models"], "entities": ["Synapse", "KKloudTarus", "Ollama", "vLLM", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug", "markdown": "https://wpnews.pro/news/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug.md", "text": "https://wpnews.pro/news/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug.txt", "jsonld": "https://wpnews.pro/news/let-an-ai-clear-out-your-false-positives-without-letting-it-hide-a-real-bug.jsonld"}}