{"slug": "vulnbench-can-llms-find-the-same-security-bugs-twice", "title": "VulnBench: Can LLMs find the same security bugs twice?", "summary": "Snyk's VulnBench benchmark found that large language models (LLMs) are not fully repeatable in security reviews: across 300 scans of 10 JavaScript projects with 6 configurations repeated 5 times, 84.8% of reference-matched findings (134 of 158) appeared in all five runs, while 49.7% of unmatched findings (80 of 161) appeared only once. The study, based on the JS 1.0 paper (arXiv:2606.15762), also found that higher session cost did not reliably improve agreement with Snyk Code's deterministic results.", "body_md": "A Snyk benchmark initiative\n\n# Can LLMs find the same bugs twice?\n\nA repeatability and Snyk-reference agreement study\n\nWe ran the same agentic security review five times against inspectable JavaScript projects to measure what recurs, what varies, and how model findings align with a deterministic Snyk Code reference set.\n\n- 300\n- scans\n- 10\n- projects\n- 6\n- configurations\n- 5\n- repetitions\n\nHeadline evidence · 5 identical reviews\n\n## Same review. Different results.\n\n300 scans · 10projects · 6 configurations\n\n**84.8%**\n\n**13.7%**\n\n**49.7%**\n\n**134 of 158** Reference-matched findings seen in all five runsInspect**22 of 161** Unmatched findings seen in all five runsInspect**80 of 161** Unmatched findings seen in only one runInspect\n\n**134 of 158** · Reference-matched findings seen in all five runs\n\n## View exact recurrence values\n\n| Finding group | Count | Share |\n|---|---|---|\n| Reference-matched findings seen in all five runs | 134 of 158 | 84.8% |\n| Unmatched findings seen in all five runs | 22 of 161 | 13.7% |\n| Unmatched findings seen in only one run | 80 of 161 | 49.7% |\n\n[Source: published JS 1.0 paper](https://arxiv.org/abs/2606.15762)\n\nLatest evidence · JS 1.0\n\n## Same code. Same prompt. Different findings.\n\nAcross five identical runs, 84.8% of reference-matched findings recurred every time. Nearly half of unmatched reports appeared only once—evidence to inspect, not dismiss.\n\nWhat the results mean in practice\n\n## An AI review is a measurement—not a verdict.\n\nRepeat the same task and the story sharpens: many reference-matched findings hold steady; AI review and deterministic SAST reveal different blind spots; and more spend does not reliably improve Snyk-reference agreement.\n\n- 01\n### Repeatability makes confidence visible\n\n134 of 158 reference-matched findings appeared in every one of five identical reviews. Repetition shows which reported patterns persist instead of relying on a single run.\n\n**Interpret with care:** A reference match measures agreement with Snyk Code, not independent ground-truth accuracy.[Open the evidence](/releases/js-1.0#repeatability) - 02\n### AI review and SAST expose different blind spots\n\nModels surfaced high-signal exploit shapes, while deterministic Snyk Code consistently enumerated repeated data-flow sinks. Their differing results are useful evidence—not a reason to declare one a universal winner.\n\n**Interpret with care:** Unmatched findings require case-level inspection before they can be classified.[Open the evidence](/releases/js-1.0#coverage) - 03\n### Paying more did not reliably improve the result\n\nIn this benchmark, higher session cost did not consistently yield higher Snyk-reference F1. Spend alone is a poor shortcut for choosing a configuration.\n\n**Interpret with care:** Cost estimates reflect the tested small fixtures and publication assumptions.[Open the evidence](/releases/js-1.0#efficiency)\n\nBenchmark anatomy\n\n## How VulnBench measures behavior\n\nRepeat the conditions, preserve the evidence, and separate observed agreement from claims the protocol cannot support.\n\n[Read the full methodology](/releases/js-1.0/methodology)\n\n- 1\n### Select inspectable projects\n\nTen small JavaScript and Express fixtures make every run and reference finding reviewable.\n\n- 2\n### Repeat the same task\n\nEach configuration sees the same code, prompt, harness, and task five times.\n\n- 3\n### Normalize findings\n\nReported issues become documented signatures suitable for recurrence analysis.\n\n- 4\n### Match the reference set\n\nThe scorer compares vulnerability type against deterministic Snyk Code findings.\n\n- 5\n### Measure behavior\n\nAgreement, recurrence, variance, coverage, cost, tokens, and duration stay distinct.\n\n- 6\n### Inspect divergence\n\nUnmatched reports remain evidence to investigate—not automatic false positives.\n\nResearch principles\n\n## Evidence before ranking\n\nVulnBench is a versioned research initiative, not a universal leaderboard. Every release defines what it measured and what it did not prove.\n\n- Transparent reference sets\n- Definitions and limitations stay visible.\n- Repeated measurement\n- Variance is a result, not a footnote.\n- Inspectable cases\n- Headline claims link toward underlying evidence.\n- Reproducible data\n- Versioned source artifacts remain downloadable.\n- Explicit limitations\n- Agreement is never relabeled as accuracy.\n\nRelease history\n\n**Snyk VulnBench JS 1.0** Published 11 June 2026 · Current\n\nFurther releases are planned. Unpublished results will not appear as speculative rankings or empty release cards.\n\n[View release catalog](/releases)\n\nPublication\n\n## Read, reproduce, and cite the work\n\nThe paper, reviewed methodology, source snapshot, and release data use stable public links.\n\nPreferred citation\n\n```\n@misc{tal2026snykvulnbenchjs10,\n  author = {Tal, Liran and Kloos, Johannes and Rudich, Arsenii and Thoemmes, Stephen and Nair, Manoj},\n  title = {Snyk VulnBench JS 1.0: Can LLMs Find the Same Bugs Twice?},\n  year = {2026},\n  url = {https://arxiv.org/abs/2606.15762}\n}\n```\n\n", "url": "https://wpnews.pro/news/vulnbench-can-llms-find-the-same-security-bugs-twice", "canonical_source": "https://vulnbench.com/", "published_at": "2026-08-20 11:27:20+00:00", "updated_at": "2026-08-20 11:45:33.896488+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-tools"], "entities": ["Snyk", "VulnBench", "Snyk Code", "JavaScript", "Express"], "alternates": {"html": "https://wpnews.pro/news/vulnbench-can-llms-find-the-same-security-bugs-twice", "markdown": "https://wpnews.pro/news/vulnbench-can-llms-find-the-same-security-bugs-twice.md", "text": "https://wpnews.pro/news/vulnbench-can-llms-find-the-same-security-bugs-twice.txt", "jsonld": "https://wpnews.pro/news/vulnbench-can-llms-find-the-same-security-bugs-twice.jsonld"}}