Last time I wrote about wiring up a security gate that blocks merges in CI with Synapse. Someone left a comment that hit the exact sore spot:
"The gate is the easy part. Two weeks in, the team turns it off because it's red over false positives every single time."
Yep. A gate that cries wolf too often dies on its own. People start hitting "merge anyway," and by the time a real vulnerability shows up, nobody's looking anymore.
The obvious 2020s fix is to hand the whole pile to an LLM and say "delete the junk." Except: the moment a security tool lets an AI delete findings, you've just built a brand new vulnerability. The model guesses wrong once, a real SQL injection quietly drops off the report, and nobody's the wiser.
So Synapse does it differently. The AI is allowed to propose that a finding is a false positive. It is never allowed to confirm that on its own, and it is never allowed to delete anything. This whole post is a real run. Terminal output at the bottom, nothing faked.
Three rules, and all three exist so you can trust the output:
1. The model only proposes. It doesn't return prose, and it doesn't return a delete command. It returns exactly one typed verdict: verdict
is one of refuted | sound | uncertain
, driver
is a closed snake_case token (not a sentence), and confidence
is 0–100. That grammar is validated in Go. The model can hallucinate a whole paragraph of reasoning and none of it reaches the report.
2. A second, different model has to agree. When the first model proposes "refuted" with confidence ≥ 75, a separate verifier model gets called to assess the same finding independently. The refutation only stands if the verifier also says "refuted" at ≥ 75. And the verifier is prompted adversarially. Its job is to
3. Nothing gets deleted. A "refuted" finding is retain-and-mark: it stays in the report, it stays evidence-sealed, it still shows up in the compliance table. It's just exempt from the gate's exit code. The worst a wrong verdict can do is let one finding skip --fail-on
. It can never make a finding disappear.
One more thing, because it matters for the cost: this is the second layer. Before any model runs, a deterministic scope classifier already strips the obvious noise: test files, fixtures, that sort of thing. The model only gets the harder calls: is this sink actually attacker-controlled? Is that interpolated value a constant or user input? Model calls cost tokens, so they only run on production-scope, first-party source findings.
It's opt-in. Four environment variables:
export SYNAPSE_FP_TRIAGE_ENABLED=true # turn on AI false-positive triage
export SYNAPSE_LLM_BASE_URL=http://localhost:20128/v1 # any OpenAI-compatible endpoint
export SYNAPSE_FP_TRIAGE_MODEL=<proposer-model> # a cheap/fast model to propose
export SYNAPSE_VERIFIER_MODEL=<different-model> # a DIFFERENT model to verify (two-model consensus)
A few notes:
SYNAPSE_LLM_BASE_URL
is a plain OpenAI-style endpoint (/v1/chat/completions
). Run it locally with Ollama or vLLM, or point it at OpenAI. Doesn't matter. Add SYNAPSE_LLM_API_KEY
if yours needs a key.SYNAPSE_VERIFIER_MODEL
(or you set it to the same model), you get single-model mode. It still works, but the real gate is the second model being Then you run the exact same scan command as always:
synapse-cli scan . --fail-on high
I built a small service with both real bugs and deliberate false positives:
payments/store.py
: a SQL query built with db.session.execute(f"... {account_id}")
where account_id
comes straight from request.args
. That's a hashlib.md5(...)
call used as a payments/settings.py
: a hardcoded AWS access key. reports/ledger.py
: another f-string SQL query, but the interpolated value is PERIOD
, a
$ synapse-cli scan . --fail-on high
Synapse SCA dogfood – payments-service
vulnerabilities: 0
findings (promoted): 7
critical risk 0.00 Hardcoded AWS access key id (payments/settings.py:4)
high risk 0.00 SQL execution uses dynamic string construction (payments/store.py:14)
high risk 0.00 Python SQLAlchemy/raw SQL uses dynamic string construction (payments/store.py:14)
medium risk 0.00 Weak hash: MD5 (payments/store.py:21)
high risk 0.00 SQL execution uses dynamic string construction (reports/ledger.py:13)
high risk 0.00 Python SQLAlchemy/raw SQL uses dynamic string construction (reports/ledger.py:13)
high risk 0.00 AWS access key ID (payments/settings.py:4)
synapse-cli: 6 finding(s) at or above high
$ echo $?
1
Gate is red over 6 findings at or above high. Two of them (the SQL in reports/ledger.py
) are false alarms you'd have to triage by hand.
The proposer and the verifier are two different models on my endpoint:
$ export SYNAPSE_FP_TRIAGE_ENABLED=true
$ export SYNAPSE_FP_TRIAGE_MODEL=gpt-5.4-mini # proposer
$ export SYNAPSE_VERIFIER_MODEL=gpt-5.4 # verifier (a DIFFERENT model)
$ synapse-cli scan . --fail-on high
synapse-cli: AI false-positive triage (gpt-5.4-mini, verified by gpt-5.4): critiqued 7 finding(s), 3 suspected false positive(s) held back from the gate
Synapse SCA dogfood – payments-service
findings (promoted): 7 # <- still 7, nothing deleted
...
compliance: Synapse AppSec Baseline v1.0 – 3/7 controls passing
[FAIL] SAB-INJ-1 No injection weaknesses ...
- SQL ... (payments/store.py:14)
- SQL ... (reports/ledger.py:13) # <- the FP still shows in compliance
...
synapse-cli: 2 suspected false positive(s) at or above high held back from the gate by AI triage (reported with a verdict; not deleted)
synapse-cli: 4 finding(s) at or above high
$ echo $?
1
Read those lines closely:
critiqued 7 finding(s), 3 suspected false positive(s) held back
: the AI looked at 7 findings and proposed 3 as false.2 ... at or above high held back ... (reported with a verdict; not deleted)
: two of those were high-severity, held back from the gate but 4 finding(s) at or above high
: the 4 real bugs still count. Gate is still exit 1
). The AI couldn't bury a real bug.findings (promoted): 7
is unchanged, and the false positives are still sitting in the compliance table. Nothing left the report.This is my favorite part. With --json
, every finding carries the AI's verdict next to it:
[critical] Hardcoded AWS access key id (settings.py:4)
verdict=sound confidence=98 suspected_fp=false driver=hardcoded_secret
[high] SQL execution dynamic string (store.py:14)
verdict=sound confidence=99 suspected_fp=false driver=confirmed_by_review
[high] SQLAlchemy raw SQL dynamic (store.py:14)
verdict=sound confidence=99 suspected_fp=false driver=dynamic_string_sql
[high] AWS access key ID (settings.py:4)
verdict=sound confidence=99 suspected_fp=false driver=literal
[medium] Weak hash: MD5 (store.py:21)
verdict=refuted confidence=96 suspected_fp=true driver=unspecified_refutation
[high] SQL execution dynamic string (ledger.py:13)
verdict=refuted confidence=99 suspected_fp=true driver=constant_literal
[high] SQLAlchemy raw SQL dynamic (ledger.py:13)
verdict=refuted confidence=98 suspected_fp=true driver=constant_literal
Look at the driver
column. For reports/ledger.py
the model worked out on its own that the interpolated value is a constant_literal
, a constant, not input. For store.py
it's sound
: real bug, left alone. And suspected_fp=true
is the only thing that exempts a finding from the gate. The key detail: because I configured a different verifier (the log literally says verified by gpt-5.4
), a finding only gets suspected_fp=true
when both models refute it at ≥ 75. Drop the verifier and these exact three refutations still have to clear a second model to stand.
End result: you're left with 4 real problems to fix instead of 6-with-2-fake mixed in. And more importantly, not one real bug got dimmed.
Right question. The whole design is built around it:
driver
is forced into a closed token grammar, verdict
has to be from a closed vocabulary, anything else is rejected. No LLM gets prose into the report path.Put another way: the worst a bad verdict can do is exempt one finding from the exit code while it stays visible in the report. Not hide a bug.
Nothing about the workflow from the last post changes. Add four environment variables to the scan step and you're done:
- name: Synapse scan (SARIF + gate on high)
env:
SYNAPSE_FP_TRIAGE_ENABLED: "true"
SYNAPSE_LLM_BASE_URL: ${{ secrets.LLM_BASE_URL }}
SYNAPSE_LLM_API_KEY: ${{ secrets.LLM_API_KEY }}
SYNAPSE_FP_TRIAGE_MODEL: "your-proposer-model"
SYNAPSE_VERIFIER_MODEL: "your-verifier-model" # different from the proposer
run: synapse-cli scan . --sarif --fail-on high > synapse.sarif
The gate stays as deterministic as before. The pile of false alarms just stops eroding the team's trust in it. And since nothing gets deleted, the SARIF you push to Code scanning is still complete; the false positives are only marked, not dropped.
One honest caveat: this is a noise filter, not a replacement for a human. It gets you looking at the 4 real problems instead of 6 muddled ones. You still have to fix those 4.
Synapse is open source (Apache-2.0): ** github.com/KKloudTarus/synapse-ce**.
driver
vocabulary, the consensus threshold: it all lives in internal/usecase/fptriage
, and it's short and easy to read.If you turn this on in your own repo, drop a comment with how many false positives the AI held back on the first run, and whether it ever came close to refuting something that turned out to be real. I'm curious.