# VulnBench: Can LLMs find the same security bugs twice?

> Source: <https://vulnbench.com/>
> Published: 2026-08-20 11:27:20+00:00

A Snyk benchmark initiative

# Can LLMs find the same bugs twice?

A repeatability and Snyk-reference agreement study

We ran the same agentic security review five times against inspectable JavaScript projects to measure what recurs, what varies, and how model findings align with a deterministic Snyk Code reference set.

- 300
- scans
- 10
- projects
- 6
- configurations
- 5
- repetitions

Headline evidence · 5 identical reviews

## Same review. Different results.

300 scans · 10projects · 6 configurations

**84.8%**

**13.7%**

**49.7%**

**134 of 158** Reference-matched findings seen in all five runsInspect**22 of 161** Unmatched findings seen in all five runsInspect**80 of 161** Unmatched findings seen in only one runInspect

**134 of 158** · Reference-matched findings seen in all five runs

## View exact recurrence values

| Finding group | Count | Share |
|---|---|---|
| Reference-matched findings seen in all five runs | 134 of 158 | 84.8% |
| Unmatched findings seen in all five runs | 22 of 161 | 13.7% |
| Unmatched findings seen in only one run | 80 of 161 | 49.7% |

[Source: published JS 1.0 paper](https://arxiv.org/abs/2606.15762)

Latest evidence · JS 1.0

## Same code. Same prompt. Different findings.

Across five identical runs, 84.8% of reference-matched findings recurred every time. Nearly half of unmatched reports appeared only once—evidence to inspect, not dismiss.

What the results mean in practice

## An AI review is a measurement—not a verdict.

Repeat the same task and the story sharpens: many reference-matched findings hold steady; AI review and deterministic SAST reveal different blind spots; and more spend does not reliably improve Snyk-reference agreement.

- 01
### Repeatability makes confidence visible

134 of 158 reference-matched findings appeared in every one of five identical reviews. Repetition shows which reported patterns persist instead of relying on a single run.

**Interpret with care:** A reference match measures agreement with Snyk Code, not independent ground-truth accuracy.[Open the evidence](/releases/js-1.0#repeatability) - 02
### AI review and SAST expose different blind spots

Models surfaced high-signal exploit shapes, while deterministic Snyk Code consistently enumerated repeated data-flow sinks. Their differing results are useful evidence—not a reason to declare one a universal winner.

**Interpret with care:** Unmatched findings require case-level inspection before they can be classified.[Open the evidence](/releases/js-1.0#coverage) - 03
### Paying more did not reliably improve the result

In this benchmark, higher session cost did not consistently yield higher Snyk-reference F1. Spend alone is a poor shortcut for choosing a configuration.

**Interpret with care:** Cost estimates reflect the tested small fixtures and publication assumptions.[Open the evidence](/releases/js-1.0#efficiency)

Benchmark anatomy

## How VulnBench measures behavior

Repeat the conditions, preserve the evidence, and separate observed agreement from claims the protocol cannot support.

[Read the full methodology](/releases/js-1.0/methodology)

- 1
### Select inspectable projects

Ten small JavaScript and Express fixtures make every run and reference finding reviewable.

- 2
### Repeat the same task

Each configuration sees the same code, prompt, harness, and task five times.

- 3
### Normalize findings

Reported issues become documented signatures suitable for recurrence analysis.

- 4
### Match the reference set

The scorer compares vulnerability type against deterministic Snyk Code findings.

- 5
### Measure behavior

Agreement, recurrence, variance, coverage, cost, tokens, and duration stay distinct.

- 6
### Inspect divergence

Unmatched reports remain evidence to investigate—not automatic false positives.

Research principles

## Evidence before ranking

VulnBench is a versioned research initiative, not a universal leaderboard. Every release defines what it measured and what it did not prove.

- Transparent reference sets
- Definitions and limitations stay visible.
- Repeated measurement
- Variance is a result, not a footnote.
- Inspectable cases
- Headline claims link toward underlying evidence.
- Reproducible data
- Versioned source artifacts remain downloadable.
- Explicit limitations
- Agreement is never relabeled as accuracy.

Release history

**Snyk VulnBench JS 1.0** Published 11 June 2026 · Current

Further releases are planned. Unpublished results will not appear as speculative rankings or empty release cards.

[View release catalog](/releases)

Publication

## Read, reproduce, and cite the work

The paper, reviewed methodology, source snapshot, and release data use stable public links.

Preferred citation

```
@misc{tal2026snykvulnbenchjs10,
  author = {Tal, Liran and Kloos, Johannes and Rudich, Arsenii and Thoemmes, Stephen and Nair, Manoj},
  title = {Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?},
  year = {2026},
  url = {https://arxiv.org/abs/2606.15762}
}
```