{"slug": "show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next", "title": "Show HN: An AI agent fixed 98% of vulnerable deps in one run, 14% in the next", "summary": "A study found that the Bomly MCP server dramatically improved the performance of AI coding agents in fixing vulnerable dependencies in a large 13-module Maven project. Claude Code's lowest completeness rose from 14% to 98% when using Bomly MCP, while Codex CLI's average runtime dropped from 13 minutes to 7.5 minutes. The results suggest that providing structured dependency and fix context via MCP can make remediation more reliable and efficient for large projects.", "body_md": "# A coding agent fixed only 14% of vulnerable advisories. Bomly MCP set a 98% floor.\n\nOn a 13-module Maven project, Bomly MCP removed Claude Code's catastrophic misses and made Codex CLI about 1.7× faster. Smaller apps did fine without it.\n\nGive a coding agent the vulnerable-dependency list and fix context over MCP, and remediating a large project stops being a gamble. That is the result of a study we ran with two leading agents — Claude Code on Opus 4.8 and Codex CLI on GPT-5.5 — on a 13-module Maven project with about 300 dependencies. Every run with the [Bomly MCP server](/mcp) connected finished at 98% complete or above, where complete means the share of fixable advisories actually resolved. What the server fixed differed by agent: Claude Code's bare runs swung between 14% and 98%, and claimed more fixes that did not hold up; Codex CLI's bare runs were already at 93–100%, but took almost twice as long. On three smaller apps, the agents did well on their own; we report that too, because it marks where the server earns its place and where it doesn't.\n\nThe large-project result in four numbers (five runs per agent and condition):\n\n| Result | Bare | Bomly MCP |\n|---|---|---|\n| Claude Code — lowest completeness | 14% | 98% |\n| Claude Code — runs with wrong claims | 4 of 5 | 2 of 5 |\n| Codex CLI — lowest completeness | 93% | 100% |\n| Codex CLI — average runtime | ~13 min | 7.5 min |\n\nNoteEverything is public: fixtures, exact prompts, raw transcripts, scoring code, and every number in this post is in\n\n[bomly-dev/bomly-agent-study]. The long-form writeup with all tables lives there too, in[REPORT.md].\n\n## What we ran\n\nTwo agents, pinned and recorded per run:\n\n| Version | Model | Reasoning | |\n|---|---|---|---|\n| Claude Code | 2.1.201 | claude-sonnet-5 (three smaller fixtures) · claude-opus-4-8 (large fixture) | high |\n| Codex CLI | 0.142.5 | gpt-5.5 | medium |\n\nEach agent ran in two conditions on each fixture:\n\n**bare**— the agent, its own tools, and open network access.** mcp**— the same setup plus a running Bomly MCP server (`bomly mcp serve`\n\n), which can return the dependency graph, the vulnerable-package list, and fix context.\n\nThe large fixture got five runs per agent and condition. The three smaller fixtures got one: after the first round came back at or near the ceiling, we dropped their remaining planned rounds and moved that budget to the large fixture.\n\nThe task prompt is identical in both conditions: find and fix the vulnerable dependencies, keep the build green. The instruction files differ by one equal-length line disclosing which tools are available. Each run starts from a fresh copy of the fixture with the answer key and scoring code stripped out, inside a pinned Docker image, and is scored mechanically afterward against a frozen ground truth: the full set of confirmed-vulnerable packages, not a hand-picked subset. [METHODOLOGY.md](https://github.com/bomly-dev/bomly-agent-study/blob/main/METHODOLOGY.md) has the details, including how the study design changed along the way and why.\n\nThree of the four fixtures are real applications vendored at old, genuinely vulnerable tags: [CTFd](https://github.com/CTFd/CTFd) 3.7.7 (Python), [Dependency-Track](https://github.com/DependencyTrack/dependency-track) 4.10.0 (single-module Maven), and — the large one — [Internet2 Grouper](https://github.com/Internet2/grouper) 4.x: a 13-module Maven build with roughly 300 resolved dependencies, 21 confirmed-vulnerable packages, and 56 fixable advisories. The fourth is a small npm app built for the study. No vulnerability anywhere is invented: every advisory is a real, published one against a real package version.\n\n## Limitations, first\n\nN=5 per cell on one large-project fixture, and N=1 per cell on the three smaller fixtures, so these are observations from our setup, not general claims. On the large fixture the vulnerable surface comes from Bomly's own resolution (the free independent scanners we used elsewhere cannot resolve transitive dependencies for multi-module Maven, and the commercial tools that can were not available to us), with hand-verified spot checks — and whether a fix actually landed is checked from the build and the diff, independently of Bomly. The two agents ran different models with different settings, so we compare each agent only to itself. And because some runs varied a lot, every chart shows every run, not means. The full list is in [LIMITATIONS.md](https://github.com/bomly-dev/bomly-agent-study/blob/main/LIMITATIONS.md).\n\n## On the smaller apps, there was little room to improve\n\nThis was supposed to be a difficulty ladder. It turned out to be a ceiling: on the npm, Python, and single-module Maven fixtures, five of the six bare runs finished at 99–100%. The exception was one Claude Code run on the Python fixture, which stopped at 56% and broke the shared build. With one run per cell here, we do not read the bare-versus-MCP differences as evidence of an effect — the useful result is that capable agents had little completeness headroom on small projects.\n\nSmall projects are within reach of a capable agent on its own: read the manifests, run `npm audit`\n\nor `pip-audit`\n\nwhere they exist, bump versions. In our setup the server added no completeness there, and we would rather say that plainly than stretch a small difference. Two side notes survived even here: three runs — two bare, one MCP — finished \"complete\" with the shared build broken (the dagger marks), and Codex CLI handled the one genuinely unfixable advisory on the Maven fixture better with the server connected — it declined the package with the right explanation, where its bare run skipped it silently.\n\nThat left the question the ladder could not answer: what happens when the project is too big to walk by hand? So we built a fourth fixture that is.\n\n## On the large project, the server set the floor\n\nGrouper is where discovery, not fixing, becomes the work. Around 300 resolved dependencies across 13 modules, no native audit command, and 56 fixable advisories spread over 21 packages that the agent first has to find.\n\nWith the server connected, **no run from either agent finished below 98%**. Without it, outcomes spread — Claude Code's bare runs landed at 91%, twice at 98%, and twice at 14%; Codex CLI's ranged 93–100%. If you run agents unattended, that floor is the number that matters: **the catastrophic run simply stopped happening**.\n\nThe 14% runs were not lazy runs. In one of them, the agent made 87 tool calls — more than any run in the study's large-project cells — and still resolved only 8 of the 56 fixable advisories. The budget went into walking the module tree by hand, not into fixing. Effort was not the problem. Knowing where to spend it was:\n\nThe fix claims also got more honest. A hallucinated fix here means the run's own report says a package was fixed, but the scored diff shows the advisory still applies — worse than a miss, because it reads as done:\n\nWithout the server, Claude Code produced at least one wrong claim in 4 of 5 runs, always on the same few awkward packages: `commons-httpclient`\n\n, `axis2`\n\n, the BouncyCastle pair, `ion-java`\n\n. With the server, **its wrong claims dropped by half — from 4 of 5 runs to 2 of 5, and from 9 affected packages to 4**. For Codex CLI it was 2 of 5 in both conditions — the server did not move that number, and we report it as is.\n\nOne thing worth spelling out, because the charts raise it: a run can be 100% complete and still contain a wrong claim. Completeness counts the 56 fixable advisories. Some packages in this fixture have no working fix, and the honest outcome there is to say so. The wrong claims in the 100%-complete runs were exactly that kind: everything fixable got fixed, and the report then also claimed a \"fix\" for a package that cannot be fixed, instead of declining it.\n\nAnd there was a speed dividend:\n\nCodex CLI finished **about 1.7× faster with the server** — its slowest MCP run beat its fastest bare run. Claude Code took similar time either way and spent the saved discovery effort on remediation instead, which is where its 98% floor came from.\n\n## What we take from this\n\nIn our setup, the Bomly MCP server mattered when discovery — not fixing — was the bottleneck. Across the ten connected runs on the large Maven fixture, **every run resolved at least 98% of the fixable advisories**. For Claude Code, the server **eliminated the 14% runs** and cut runs with wrong claims from 4 of 5 to 2 of 5. For Codex CLI, completeness was already high; the win was speed, with the average run dropping **from about 13 minutes to 7.5 minutes**.\n\nOne large fixture is not enough to pin down a general size threshold. What the result suggests is that whole-graph context helps once the dependency tree is hard to enumerate by hand — big, many modules, no single audit command.\n\nThe honest other half: small projects did not need the server. A good agent handles known-CVE fixes there on its own.\n\n## The study fixed four Bomly bugs\n\nRunning our own tool underneath real agents for two weeks found four real bugs in it, which is a result we did not plan for:\n\n- a system-installed\n`pip-audit`\n\npolluted Bomly's Python interpreter detection ([#237](https://github.com/bomly-dev/bomly-cli/issues/237)) - ANSI color codes in Maven's\n`dependency:tree`\n\noutput made every transitive Maven dependency invisible ([#243](https://github.com/bomly-dev/bomly-cli/issues/243)) - oversized MCP responses truncated half the scan calls (\n[#245](https://github.com/bomly-dev/bomly-cli/issues/245)) - the Maven detector could not scan any multi-module reactor at all (\n[#252](https://github.com/bomly-dev/bomly-cli/pull/252)— the large-fixture result only exists because of this fix)\n\n## FAQ\n\n### Does the Bomly MCP server help coding agents fix vulnerable dependencies?\n\nYes — particularly on large projects. On a 13-module Maven project with about 300 dependencies and no native audit tool, every MCP-connected run resolved at least 98% of the fixable advisories. Without the server, Claude Code's runs ranged from 14% to 98%, while Codex CLI's stayed between 93% and 100%. On three smaller apps, capable agents did well on their own.\n\n### Does the Bomly MCP server make agents faster?\n\nIn our setup, Codex CLI was about 1.7× faster with the server on the large project. Claude Code spent similar time either way and converted the saved discovery work into more complete, more reliable remediation.\n\n### How can I reproduce the study?\n\nEverything is in [bomly-dev/bomly-agent-study](https://github.com/bomly-dev/bomly-agent-study): fixtures, prompts, harness, transcripts, and scoring. `make verify-only`\n\nre-scores any published run with no API key.\n\nTo set this up for your own agent:\n\n[Try Bomly MCP with your coding agent](/mcp)\n\nPrefer a walkthrough first? Follow the step-by-step guide: [make your coding agent dependency-aware](/blog/make-your-coding-agent-dependency-aware).", "url": "https://wpnews.pro/news/show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next", "canonical_source": "https://bomly.dev/blog/coding-agents-with-and-without-bomly-mcp", "published_at": "2026-07-15 14:07:42+00:00", "updated_at": "2026-07-15 14:17:58.771878+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-infrastructure"], "entities": ["Bomly MCP", "Claude Code", "Codex CLI", "OpenAI", "Anthropic", "Internet2 Grouper", "CTFd", "Dependency-Track"], "alternates": {"html": "https://wpnews.pro/news/show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next", "markdown": "https://wpnews.pro/news/show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next.md", "text": "https://wpnews.pro/news/show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next.txt", "jsonld": "https://wpnews.pro/news/show-hn-an-ai-agent-fixed-98-of-vulnerable-deps-in-one-run-14-in-the-next.jsonld"}}