{"slug": "vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build", "title": "VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?", "summary": "Snyk's new VulnBump benchmark found that on the same Claude Sonnet 5.5 model, Snyk's Evo Remediation Agent passed 90.2% of 307 real security-driven dependency upgrades, versus 77.5% for Anthropic's Claude Code with Snyk's recommended upgrade and scan and 59.6% for Claude Code unaided — a gap of more than 30 percentage points. The no-AI package manager baseline, which only applies Snyk's recommended version, passed 46.9%. Snyk said all three agents kept the build green about 93% of the time, but removing the vulnerability was the hard part: unaided Claude Code falsely claimed fixes in 70 of its 124 failures, and only about 10% of the 459 million vulnerability findings Snyk Open Source recorded across roughly 300,000 organizations over six months in 2026 had been resolved despite 94% having a known fixed version.", "body_md": "# VulnBump: Can AI Agents Fix Vulnerable Dependencies Without Breaking the Build?\n\nOctober 15, 2026\n\n0 mins read\nCan AI agents fix vulnerable dependencies without breaking the build? The fix itself is rarely the mystery. Over six months in 2026, Snyk Open Source recorded 459 million vulnerability findings across ~300,000 organizations. 94% of these vulnerabilities had a known fixed version, yet only about 10% had been resolved.\n\nThat happens because the hard part is upgrading safely: most vulnerable packages are transitively imported, about a quarter of fixes require a major version bump, and even minor and patch releases, which are supposed to be backward compatible, often contain breaking changes.\n\n[**__VulnBump__**](https://www.vulnbump.com/), new research we published at Snyk, puts AI agents to the test on 307 real upgrades of vulnerable dependencies. An upgrade counts as a **PASS** only if the vulnerabilities are gone and the project still installs, builds, and passes its tests.\n\n**What's new:** to our knowledge, VulnBump is the first benchmark for AI agents built around security-driven upgrades: every task starts from real vulnerabilities that Snyk found in a real project, across six package managers. Earlier upgrade benchmarks check only for breakage, not whether the vulnerability is gone.\n\n**So, can AI agents fix vulnerable dependencies without breaking the build?** Mostly, yes, but the model alone doesn't decide it. On the same model, Claude Sonnet 5.5, Snyk's Evo Remediation Agent passed 90.2% of upgrades. Anthropic's Claude Code passed 77.5% when given Snyk's recommended upgrade and scan (Claude Code + Snyk scan), and 59.6% with no Snyk data (Claude Code unaided). That's a gap of more than 30 percentage points. For reference, the package manager bump, a baseline with no AI that only applies Snyk's recommended version, already passed 46.9%.\n\n*Key results on VulnBump*\n\nKeeping the build green wasn't the hard part: all three agents did that about 93% of the time. Removing the vulnerability was. In 70 of its 124 failures, Claude Code, unaided, said it had fixed vulnerabilities that a rescan still found.\n\nOur findings echo [__VulnBench__](https://snyk.io/blog/snyk-vulnbench-js-1-0-llm-security-review-repeatability/), our June benchmark of AI security review, which found that Claude and Snyk Code miss different vulnerabilities and concluded that the two work best combined. VulnBump tests a different task, upgrading vulnerable open source dependencies, and points the same way: with Snyk's recommended version and scan, Claude Code's PASS rate rose by 17.9 points. Evo Remediation Agent shows how much further a purpose-built agent can go: 12.7 points beyond Claude Code with the same Snyk scan. VulnBump also follows up on [__the remediation results we shared in May__](https://snyk.io/blog/snyk-remediation-agent-in-the-cli/), now with a newer model and six package managers.\n\n## How we built VulnBump\n\n**We started from real projects, matched to production.** We collected 767 candidate projects from GitHub, including popular open source projects and repositories that pin known-vulnerable packages. Each had to install, build, and pass more than 85% of its tests before any change, and Snyk had to find at least one vulnerability. From the 99 that qualified, we picked 43 that match Snyk production data for ecosystem, vulnerability severity, direct vs transitive dependencies, and size of version jump.\n\n*VulnBump dataset comparison with Snyk Open Source production data*\n\n**We turned each recommended upgrade into a task.** Each task asks the agent to apply one upgrade that Snyk recommends: one package, moved to a version that fixes its vulnerabilities. Every task starts from a fresh copy of the project and is scored on its own. The 43 repositories yield 307 tasks covering 2,066 fixable vulnerabilities, across Maven, Gradle, npm, Yarn, pip, and Poetry, the six package managers behind about 92% of production findings.\n\n**We added a verifier that didn't trust the agent.** When the agent finished, we copied its changes into a clean container and checked them from scratch. There, an attempt counted as a PASS only if it cleared all four *gates*:\n\n1. **The dependencies install** with the command CI would run.\n2. **A new Snyk scan** confirms the vulnerabilities are gone.\n3. **The project builds.**\n4. **Every test that passed before still passes.**\n\n*The VulnBump pipeline*\n\n**The harnesses we tested.** All three agents ran on Claude Sonnet 5.5, with a limit of 110 turns per upgrade, in the same container for each language:\n\n- **Evo Remediation Agent** is Snyk’s agentic remediation engine. It uses Snyk intelligence and can run its own scan, build, and tests, and repair errors before finishing. We ran it in auto-approve mode, so it applied every fix, including upgrades Snyk rates as likely to break the build.\n- **Claude Code Snyk scan** is Anthropic's general-purpose coding agent, given Snyk's recommendation for the upgrade: the package to upgrade, the target version to move to, and the vulnerabilities it fixes. It can also run a Snyk Open Source scan itself to check its work.\n- **Claude Code unaided** gets the same task without the target version or any Snyk access. It has to determine which version fixes the vulnerabilities on its own, for example, by searching vulnerability databases on the web.\n- **The package manager bump** is the non-agentic baseline: it applies Snyk's recommended version with the project's own package manager and changes nothing else.\n\nHow the repositories were chosen, the exact prompts and tools each agent had, and the leakage controls are all in the [__research paper: VulnBump: A Benchmark for Fixing Vulnerable Dependencies in Real Projects While Keeping the Build and Tests Green__](https://arxiv.org/abs/2610.17374), along with the limitations: we tested one model, ran each upgrade once, and used Snyk Open Source as the judge, which can only count against Claude Code unaided, since the other two agents work from the same Snyk findings. We also didn't penalize new vulnerabilities that an upgrade pulled in.\n\n## Result 1: The same model, with three different outcomes\n\nEvo Remediation Agent passed every gate on 90.2% of upgrades (and fixed 95.9% of the 2,066 vulnerability instances). That's 12.7 points more than Claude Code + Snyk scan, roughly one more working fix in every eight upgrades, and 30.6 more than Claude Code unaided.\n\nAnd the ranking doesn't just hinge on a few large repositories: averaged per repository, PASS rates were 93.1% for Evo Remediation Agent, 80.1% for Claude Code + Snyk scan, 68.0% for Claude Code unaided, and 51.3% for the package manager bump.\n\n## Key finding:\n\nEvo Remediation Agent fixes 9 in 10 vulnerability-driven upgrades end-to-end.\n\n## Result 2: Agents need to know which version is actually safe\n\nOn the same model, giving Claude Code Snyk's scan and recommended upgrade raised its PASS rate by 17.9 points, from 59.6% to 77.5%.\n\nThe clearest difference is which version it picks. Unaided, Claude Code settled on a version Snyk still flags 64 times. With Snyk's scan, that dropped to 13. Vulnerability databases don't always agree on which version fixes a vulnerability, so a version that looks safe in one source can still be flagged in another.\n\nSnyk's guidance matters most for transitive vulnerabilities, in packages the project doesn't declare directly. Counting only upgrades that passed every gate, it more than doubled the share Claude Code fixed, from 28% to 59%.\n\n## Key finding:\n\nSnyk's scan and target version raised Claude Code's PASS rate by 17.9 points and more than doubled the transitive vulnerabilities it fixed.\n\n## Result 3: Snyk's advice alone solves about half, and then the agents separate on the rest\n\nThe package manager bump splits the 307 upgrades in two: On the 144 it passes, Snyk's recommended version is all it takes, and an agent can only keep or lose an easy fix. The other 163 need more than a version bump, such as adapting code to breaking changes or resolving dependency conflicts.\n\n*Agents on the 144 upgrades Snyk's advice solves, and on the other 163*\n\nWith Snyk's guidance, agents rarely break what the bump solves, while Claude Code unaided regressed on 20 of the 144, in 15 of them by picking a version Snyk still flags. The real difference becomes evident on the *hard half*: with the other 163 upgrades, Evo Remediation Agent passed 82.2%, Claude Code + Snyk scan passed 61.3%, and Claude Code by itself passed 36.2%.\n\n## Key finding:\n\nOf the 163 upgrades that need more than a version bump, Evo Remediation Agent passed 82.2%, against 61.3% for Claude Code + Snyk scan and 36.2% for Claude Code unaided.\n\n## Result 4: A green build can hide an unfixed vulnerability\n\nThe easiest way to keep a build green is to leave the vulnerability in place, and that's mostly where we saw Claude Code failing. The rescan was the first gate to fail in 107 of Claude Code’s unaided 124 failures, 48 of Claude Code + Snyk scan's 69 failures, against 9 of Evo Remediation Agent's 30.\n\n*First failed verification gate for every failed attempt*\n\n**Evo Remediation Agent's failures are visible (and that's not a bad thing)**. 21 of its 30 failures broke the build or tests, mostly on major upgrades that other agents often avoided. **A loud failure is much safer than a silent one**: when one upgrade does go wrong, the project's own build and tests show it, where a developer can actually see it, and eventually act upon it.\n\n## Key finding:\n\nGeneral-purpose agents fail silently; Evo Remediation Agent fails where CI catches it.\n\n## Result 5: Knowing the target version isn't enough\n\nSnyk's scan gets Claude Code a long way, but getting the upgrade right takes more than that: with identical findings and targets, Evo Remediation Agent passed 39 more upgrades than Claude Code Snyk scan, 23 of them major jumps.\n\nEvo Remediation Agent combines Snyk's intelligence with a loop in which it builds, tests, and repairs the project before it finishes. While our setup can't isolate which component makes the difference, it does show how far the system around a model can impact the result, which is consistent with [__DepBench__](https://benchmark-radar.org/benchmarks/claire-radar-2608-30300/), where changing only the harness changed results by up to 16.3 points.\n\n## Key finding:\n\nSame model, same Snyk findings: Evo Remediation Agent still passes 12.7 points more.\n\n## Result 6: Major jumps and transitive dependencies remain the hardest tasks\n\nAll agents still struggle with major framework migrations: 23 upgrades, mostly Spring, Spring Boot, and Dropwizard migrations, *defeated* all three agents. \n\nPASS rates fell to 81% from 96% for Evo Remediation Agent, to 61% from 88% for Claude Code + Snyk scan, and to 42% from 70% for Claude Code unaided, respectively, on major vs non-major upgrades. Major upgrades are also where code has to change: about half of Evo Remediation Agent's major version PASSes edited the application source, compared to 5% of its non-major PASSes.\n\n*Upgrades passing every gate, by the size of the version jump*\n\nTransitive vulnerabilities, where the vulnerable package isn't declared in the manifest, make up 70% of fixable instances and are harder for every harness tested. Counting only fixes from attempts that passed every gate, Evo Remediation Agent fixed 73% of transitive instances, Claude Code + Snyk scan 59%, and Claude Code unaided 28%, against 91%, 71%, and 62% respectively of direct ones.\n\n*PASS rate by dependency type (direct vs transitive)*\n\n## Key finding:\n\nEvo Remediation Agent leads on the hardest upgrades, majors, and transitive dependencies, but major framework migrations still beat every agent.\n\n## How does VulnBump fit with VulnBench?\n\n**VulnBench** evaluates how AI coding agents find vulnerabilities in the code you write: how consistently they detect the same issues across repeated runs, and how closely their findings match a deterministic SAST baseline. **VulnBump** benchmarks whether they can fix vulnerabilities in the code you depend on: upgrading vulnerable open-source dependencies without breaking the build or tests.\n\nWhen you read the two together, they point the same way: models bring reasoning that deterministic tools just don't have. On their own, they report different findings across runs and often stop short of a fix. Across the platform, we put the shared lesson more simply: **the generator cannot be the validator**.\n\n## What does this mean for teams handing security work to agents?\n\n- **Judge a fix by the evidence the agent didn't produce:** An agent's report of its own fix is a**claim** , while a rescan, a build, and a test run become**evidence** . If an agent is fixing your backlog, hold it to the same checks.\n- **Score the vulnerability and the build together:** Either one alone hides a failure. Breakage-only checks miss fixes that never happened, while vulnerability-only checks miss the fix that broke the project.\n- **Give the agent the security data:** Snyk's scan and recommended upgrade were worth 17.9 points to a general-purpose agent, on the same model.\n\nEvo Remediation Agent is Snyk's agentic remediation engine, available through the Snyk CLI and as a cloud service. It's one of the agents in Evo Agentic AppSec, which launches today.\n\n## Run your agent through VulnBump\n\nThe VulnBump harness and all 43 evaluation cases are open-sourced at [__github.com/snyk-labs/snyk-vulnbump__](https://github.com/snyk-labs/snyk-vulnbump).\n\nThe full methodology, failure analysis, and limits are in the [__research paper__](https://arxiv.org/abs/2610.17374). If you're building a remediation agent yourself, run it through VulnBump and let us know what you find.\n\n__VulnBump__ *was built by Gabija Pasiunaite, Ryan McMorrow, Brian Clark, Liran Tal, and Luke Tu.*\n\nRESEARCH", "url": "https://wpnews.pro/news/vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build", "canonical_source": "https://snyk.io/blog/vulnbump/", "published_at": "2026-10-15 04:00:00+00:00", "updated_at": "2026-10-15 11:18:18.192780+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-tools", "developer-tools", "ai-research"], "entities": ["Snyk", "VulnBump", "Snyk Evo Remediation Agent", "Claude Code", "Anthropic", "Claude Sonnet 5.5", "Snyk Open Source", "VulnBench"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build", "markdown": "https://wpnews.pro/news/vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build.md", "text": "https://wpnews.pro/news/vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build.txt", "jsonld": "https://wpnews.pro/news/vulnbump-can-ai-agents-fix-vulnerable-dependencies-without-breaking-the-build.jsonld"}}