Benchmarking LLMs on SAST Triage

We benchmarked 15 LLMs on 142 real security findings to triage SAST false positives. Kimi K3 led on accuracy, with one caveat that matters for security.

SAST tools would rather flag ten harmless lines than miss one real vulnerability. We built an LLM triage layer that knows how to say "I'm not sure."

By Anže Pečar, Founding Engineer & Tim Olshansky, Co-Founder, Fencer

SAST tools are tuned for high recall (flag ten harmless lines rather than miss the one real vulnerability). The result is predictable: teams learn to ignore the scanner, and the one finding that matters gets buried with everything else.

At Fencer we run an LLM triage layer over our scanners' output: each finding gets read in context and classified as a real vulnerability or a false positive. We benchmarked fifteen models: Claude Opus 5, Opus 4.8, Sonnet 5, and Haiku 4.5; GPT-5.6 Sol, Terra, and Luna; Kimi K3; and a stack of open-weight models, all against 142 findings our engineers had already reviewed.

Kimi K3 came first on accuracy, clearing more false positives than any Claude or GPT model we tested. It's also the only model near the top of the table that dismissed a real vulnerability as a false positive. In security triage, that's the one mistake you can't shrug off, and it's why we're not switching to it today.

Hand-labeling a few dozen findings is quick but noisy. Instead we derived ground truth from what actually happened in production: engineers' thumbs-up/down on past triage verdicts, findings dismissed as false positives, and findings that got fixed. 142 findings total: 11 confirmed vulnerabilities and 131 false positives.

That ratio reflects reality. Real SAST output is overwhelmingly noise, so a triage model has two jobs: clear the false positives (131 chances to be useful) and never quietly drop a real vulnerability (11 chances to do real harm).

Following the LLM4SA approach, each finding goes to the model with its sink, callers, callees, and taint sources, the same evidence a human reviewer would read before making a call. Each finding is classified three times; majority vote wins. The model is instructed to be conservative: when it can't rule the vulnerability out, it returns uncertain

and the finding goes to a human.

We score three outcomes on the confirmed vulnerabilities: caught, uncertain (escalated to a human), and dismissed (silently closed as a false positive). The distinction between uncertain and dismissed is the whole point of the design.

*Opus 5 scored on the 131 of 142 findings it would answer (see below).

The headline isn't K3's top spot, it's the gap to its predecessor. Kimi K2.5 finished twelfth at 73%, catching 5 of 11 real vulnerabilities and returning uncertain on 32 findings. K3 sits at 92% with 122 of 131 false positives cleared and only 5 uncertain. Largest generational jump we measured from any vendor.

It wins on the axis that matters most for noise: 122 of 131, two ahead of Opus 4.8 and six ahead of Sonnet 5. And it's decisive: only 5 uncertain, where most mid-tier models punt 20 to 33 findings back to a human.

Then there's the caveat. On one finding, an HTML fragment that writes a fetched response straight into innerHTML

, K3 was the only model in the entire lineup to rule it safe. All fourteen others flagged it or escalated it. K3 dismissed it unanimously across all three runs, and its reasoning shows exactly how:

"the response body istrustedsame-origin server output, not direct attacker input; any user data within itmust be escaped by the server-side renderer"

K3 didn't miss the sink. It spotted it, then talked itself into assuming a protection it couldn't verify. "The server must be escaping this" is an assumption, not evidence.

Opus 4.8 and Sonnet 5 dismissed zero real vulnerabilities. Sonnet 5 caught all 11. When those models can't confirm that data is safe, they flag or escalate. K3 sometimes decides on its own.

On a queue that's 92% noise, K3's extra noise-clearing saves real review time. But one silently-closed vulnerability costs more than a hundred extra false positives on a reviewer's plate. We treat "vulnerabilities dismissed" as a gate, not a column to average away.

Opus 5 landed on AWS days before we ran this. We expected it to compete for first. Instead it refused 11 of 142 findings outright: content-filtered, no verdict.

The blocked prompts weren't exotic: injection sinks, command-execution paths, the same material every other model analyzed without complaint. A classifier somewhere reads "here is a potential SQL injection" as dual-use security content and stops.

On the 131 findings it would answer, Opus 5 had the highest accuracy in the test and zero dismissed vulnerabilities. But 8% of the set never got scored, and we can't know whether those were the easy findings or the ones that mattered. A triage model that silently skips findings isn't shippable, regardless of how well it scores on the rest.

Worth flagging if you're building in this space: safety tuning that makes a general-purpose assistant deployable can make it unusable for defensive security work. Analyzing a vulnerability and exploiting one look the same to a classifier. We hit this harder on another frontier Claude model whose docs explicitly state it blocks dual-use cybersecurity content. Test for refusals explicitly: they show up as errors in your logs, not as bad scores on your leaderboard, so they're easy to miss until production.

data-*

attribute written into innerHTML

, Sol sees no sanitizer and flags it. Opus 4.8, same finding, same prompt, reasons that the value is developer-authored markup rather than runtime user input, notices the call graph is spurious, and dismisses it. One model asks "is there a dangerous sink here," the other asks "can an attacker actually reach it."Open models are no longer the compromise option. The K2.5 to K3 jump was larger than any gap between the US frontier models we tested.

Rank on the failure that hurts. Accuracy averages a dismissed vulnerability and a cleared false positive as if they're equivalent. They're not. Separate "dismissed" from "uncertain" and the ordering changes.

Judgment cuts both ways. The reasoning that makes a model good at triage (can an attacker actually reach this sink?) is the same reasoning that lets it invent safeguards it can't see.

Content filtering is a deployment risk. Our most capable model was disqualified by its refusals, not its accuracy. Measure refusals before you build on a model.

We expected to crown the smartest model. We came out with something more useful: the model you want for high-volume triage is the one that reasons about reachability, escalates when it isn't sure, and looks at every finding you send it.