Let an AI clear out your false positives without letting it hide a real bug

Synapse, a security gate tool, introduces AI-powered false-positive triage that never deletes findings. The system uses a two-model consensus: a proposer model suggests a false-positive verdict, and a separate verifier model must independently agree. Refuted findings are retained and marked but exempt from the gate's exit code, preventing AI from hiding real bugs.

Last time I wrote about wiring up a security gate that blocks merges in CI with Synapse https://github.com/KKloudTarus/synapse-ce . Someone left a comment that hit the exact sore spot: "The gate is the easy part. Two weeks in, the team turns it off because it's red over false positives every single time." Yep. A gate that cries wolf too often dies on its own. People start hitting "merge anyway," and by the time a real vulnerability shows up, nobody's looking anymore. The obvious 2020s fix is to hand the whole pile to an LLM and say "delete the junk." Except: the moment a security tool lets an AI delete findings, you've just built a brand new vulnerability. The model guesses wrong once, a real SQL injection quietly drops off the report, and nobody's the wiser. So Synapse does it differently. The AI is allowed to propose that a finding is a false positive. It is never allowed to confirm that on its own, and it is never allowed to delete anything. This whole post is a real run. Terminal output at the bottom, nothing faked. Three rules, and all three exist so you can trust the output: 1. The model only proposes. It doesn't return prose, and it doesn't return a delete command. It returns exactly one typed verdict: verdict is one of refuted | sound | uncertain , driver is a closed snake case token not a sentence , and confidence is 0–100. That grammar is validated in Go. The model can hallucinate a whole paragraph of reasoning and none of it reaches the report. 2. A second, different model has to agree. When the first model proposes "refuted" with confidence ≥ 75, a separate verifier model gets called to assess the same finding independently. The refutation only stands if the verifier also says "refuted" at ≥ 75. And the verifier is prompted adversarially. Its job is to 3. Nothing gets deleted. A "refuted" finding is retain-and-mark: it stays in the report, it stays evidence-sealed, it still shows up in the compliance table. It's just exempt from the gate's exit code. The worst a wrong verdict can do is let one finding skip --fail-on . It can never make a finding disappear. One more thing, because it matters for the cost: this is the second layer. Before any model runs, a deterministic scope classifier already strips the obvious noise: test files, fixtures, that sort of thing. The model only gets the harder calls: is this sink actually attacker-controlled? Is that interpolated value a constant or user input? Model calls cost tokens, so they only run on production-scope, first-party source findings. It's opt-in. Four environment variables: export SYNAPSE FP TRIAGE ENABLED=true turn on AI false-positive triage export SYNAPSE LLM BASE URL=http://localhost:20128/v1 any OpenAI-compatible endpoint export SYNAPSE FP TRIAGE MODEL=<proposer-model a cheap/fast model to propose export SYNAPSE VERIFIER MODEL=<different-model a DIFFERENT model to verify two-model consensus A few notes: SYNAPSE LLM BASE URL is a plain OpenAI-style endpoint /v1/chat/completions . Run it locally with Ollama or vLLM, or point it at OpenAI. Doesn't matter. Add SYNAPSE LLM API KEY if yours needs a key. SYNAPSE VERIFIER MODEL or you set it to the same model , you get single-model mode. It still works, but the real gate is the second model being Then you run the exact same scan command as always: synapse-cli scan . --fail-on high I built a small service with both real bugs and deliberate false positives : payments/store.py : a SQL query built with db.session.execute f"... {account id}" where account id comes straight from request.args . That's a hashlib.md5 ... call used as a payments/settings.py : a hardcoded AWS access key. reports/ledger.py : another f-string SQL query, but the interpolated value is PERIOD , a bash $ synapse-cli scan . --fail-on high Synapse SCA dogfood – payments-service vulnerabilities: 0 findings promoted : 7 critical risk 0.00 Hardcoded AWS access key id payments/settings.py:4 high risk 0.00 SQL execution uses dynamic string construction payments/store.py:14 high risk 0.00 Python SQLAlchemy/raw SQL uses dynamic string construction payments/store.py:14 medium risk 0.00 Weak hash: MD5 payments/store.py:21 high risk 0.00 SQL execution uses dynamic string construction reports/ledger.py:13 high risk 0.00 Python SQLAlchemy/raw SQL uses dynamic string construction reports/ledger.py:13 high risk 0.00 AWS access key ID payments/settings.py:4 synapse-cli: 6 finding s at or above high $ echo $? 1 Gate is red over 6 findings at or above high . Two of them the SQL in reports/ledger.py are false alarms you'd have to triage by hand. The proposer and the verifier are two different models on my endpoint: bash $ export SYNAPSE FP TRIAGE ENABLED=true $ export SYNAPSE FP TRIAGE MODEL=gpt-5.4-mini proposer $ export SYNAPSE VERIFIER MODEL=gpt-5.4 verifier a DIFFERENT model $ synapse-cli scan . --fail-on high synapse-cli: AI false-positive triage gpt-5.4-mini, verified by gpt-5.4 : critiqued 7 finding s , 3 suspected false positive s held back from the gate Synapse SCA dogfood – payments-service findings promoted : 7 <- still 7, nothing deleted ... compliance: Synapse AppSec Baseline v1.0 – 3/7 controls passing FAIL SAB-INJ-1 No injection weaknesses ... - SQL ... payments/store.py:14 - SQL ... reports/ledger.py:13 <- the FP still shows in compliance ... synapse-cli: 2 suspected false positive s at or above high held back from the gate by AI triage reported with a verdict; not deleted synapse-cli: 4 finding s at or above high $ echo $? 1 Read those lines closely: critiqued 7 finding s , 3 suspected false positive s held back : the AI looked at 7 findings and proposed 3 as false. 2 ... at or above high held back ... reported with a verdict; not deleted : two of those were high-severity, held back from the gate but 4 finding s at or above high : the 4 real bugs still count. Gate is still exit 1 . The AI couldn't bury a real bug. findings promoted : 7 is unchanged, and the false positives are still sitting in the compliance table. Nothing left the report.This is my favorite part. With --json , every finding carries the AI's verdict next to it: critical Hardcoded AWS access key id settings.py:4 verdict=sound confidence=98 suspected fp=false driver=hardcoded secret high SQL execution dynamic string store.py:14 verdict=sound confidence=99 suspected fp=false driver=confirmed by review high SQLAlchemy raw SQL dynamic store.py:14 verdict=sound confidence=99 suspected fp=false driver=dynamic string sql high AWS access key ID settings.py:4 verdict=sound confidence=99 suspected fp=false driver=literal medium Weak hash: MD5 store.py:21 verdict=refuted confidence=96 suspected fp=true driver=unspecified refutation high SQL execution dynamic string ledger.py:13 verdict=refuted confidence=99 suspected fp=true driver=constant literal high SQLAlchemy raw SQL dynamic ledger.py:13 verdict=refuted confidence=98 suspected fp=true driver=constant literal Look at the driver column. For reports/ledger.py the model worked out on its own that the interpolated value is a constant literal , a constant, not input. For store.py it's sound : real bug, left alone. And suspected fp=true is the only thing that exempts a finding from the gate. The key detail: because I configured a different verifier the log literally says verified by gpt-5.4 , a finding only gets suspected fp=true when both models refute it at ≥ 75. Drop the verifier and these exact three refutations still have to clear a second model to stand. End result: you're left with 4 real problems to fix instead of 6-with-2-fake mixed in. And more importantly, not one real bug got dimmed. Right question. The whole design is built around it: driver is forced into a closed token grammar, verdict has to be from a closed vocabulary, anything else is rejected. No LLM gets prose into the report path.Put another way: the worst a bad verdict can do is exempt one finding from the exit code while it stays visible in the report. Not hide a bug. Nothing about the workflow from the last post changes. Add four environment variables to the scan step and you're done: - name: Synapse scan SARIF + gate on high env: SYNAPSE FP TRIAGE ENABLED: "true" SYNAPSE LLM BASE URL: ${{ secrets.LLM BASE URL }} SYNAPSE LLM API KEY: ${{ secrets.LLM API KEY }} SYNAPSE FP TRIAGE MODEL: "your-proposer-model" SYNAPSE VERIFIER MODEL: "your-verifier-model" different from the proposer run: synapse-cli scan . --sarif --fail-on high synapse.sarif The gate stays as deterministic as before. The pile of false alarms just stops eroding the team's trust in it. And since nothing gets deleted, the SARIF you push to Code scanning is still complete; the false positives are only marked, not dropped. One honest caveat: this is a noise filter, not a replacement for a human. It gets you looking at the 4 real problems instead of 6 muddled ones. You still have to fix those 4. Synapse is open source Apache-2.0 : github.com/KKloudTarus/synapse-ce . driver vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage , and it's short and easy to read.If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.