On a 13-module Maven project, Bomly MCP removed Claude Code's catastrophic misses and made Codex CLI about 1.7× faster. Smaller apps did fine without it.
Give a coding agent the vulnerable-dependency list and fix context over MCP, and remediating a large project stops being a gamble. That is the result of a study we ran with two leading agents — Claude Code on Opus 4.8 and Codex CLI on GPT-5.5 — on a 13-module Maven project with about 300 dependencies. Every run with the Bomly MCP server connected finished at 98% complete or above, where complete means the share of fixable advisories actually resolved. What the server fixed differed by agent: Claude Code's bare runs swung between 14% and 98%, and claimed more fixes that did not hold up; Codex CLI's bare runs were already at 93–100%, but took almost twice as long. On three smaller apps, the agents did well on their own; we report that too, because it marks where the server earns its place and where it doesn't.
The large-project result in four numbers (five runs per agent and condition):
| Result | Bare | Bomly MCP |
|---|---|---|
| Claude Code — lowest completeness | 14% | 98% |
| Claude Code — runs with wrong claims | 4 of 5 | 2 of 5 |
| Codex CLI — lowest completeness | 93% | 100% |
| Codex CLI — average runtime | ~13 min | 7.5 min |
NoteEverything is public: fixtures, exact prompts, raw transcripts, scoring code, and every number in this post is in
[bomly-dev/bomly-agent-study]. The long-form writeup with all tables lives there too, in[REPORT.md].
What we ran #
Two agents, pinned and recorded per run:
| Version | Model | Reasoning | |
|---|---|---|---|
| Claude Code | 2.1.201 | claude-sonnet-5 (three smaller fixtures) · claude-opus-4-8 (large fixture) | high |
| Codex CLI | 0.142.5 | gpt-5.5 | medium |
Each agent ran in two conditions on each fixture:
bare— the agent, its own tools, and open network access.** mcp**— the same setup plus a running Bomly MCP server (bomly mcp serve
), which can return the dependency graph, the vulnerable-package list, and fix context.
The large fixture got five runs per agent and condition. The three smaller fixtures got one: after the first round came back at or near the ceiling, we dropped their remaining planned rounds and moved that budget to the large fixture.
The task prompt is identical in both conditions: find and fix the vulnerable dependencies, keep the build green. The instruction files differ by one equal-length line disclosing which tools are available. Each run starts from a fresh copy of the fixture with the answer key and scoring code stripped out, inside a pinned Docker image, and is scored mechanically afterward against a frozen ground truth: the full set of confirmed-vulnerable packages, not a hand-picked subset. METHODOLOGY.md has the details, including how the study design changed along the way and why.
Three of the four fixtures are real applications vendored at old, genuinely vulnerable tags: CTFd 3.7.7 (Python), Dependency-Track 4.10.0 (single-module Maven), and — the large one — Internet2 Grouper 4.x: a 13-module Maven build with roughly 300 resolved dependencies, 21 confirmed-vulnerable packages, and 56 fixable advisories. The fourth is a small npm app built for the study. No vulnerability anywhere is invented: every advisory is a real, published one against a real package version.
Limitations, first #
N=5 per cell on one large-project fixture, and N=1 per cell on the three smaller fixtures, so these are observations from our setup, not general claims. On the large fixture the vulnerable surface comes from Bomly's own resolution (the free independent scanners we used elsewhere cannot resolve transitive dependencies for multi-module Maven, and the commercial tools that can were not available to us), with hand-verified spot checks — and whether a fix actually landed is checked from the build and the diff, independently of Bomly. The two agents ran different models with different settings, so we compare each agent only to itself. And because some runs varied a lot, every chart shows every run, not means. The full list is in LIMITATIONS.md.
On the smaller apps, there was little room to improve #
This was supposed to be a difficulty ladder. It turned out to be a ceiling: on the npm, Python, and single-module Maven fixtures, five of the six bare runs finished at 99–100%. The exception was one Claude Code run on the Python fixture, which stopped at 56% and broke the shared build. With one run per cell here, we do not read the bare-versus-MCP differences as evidence of an effect — the useful result is that capable agents had little completeness headroom on small projects.
Small projects are within reach of a capable agent on its own: read the manifests, run npm audit
or pip-audit
where they exist, bump versions. In our setup the server added no completeness there, and we would rather say that plainly than stretch a small difference. Two side notes survived even here: three runs — two bare, one MCP — finished "complete" with the shared build broken (the dagger marks), and Codex CLI handled the one genuinely unfixable advisory on the Maven fixture better with the server connected — it declined the package with the right explanation, where its bare run skipped it silently.
That left the question the ladder could not answer: what happens when the project is too big to walk by hand? So we built a fourth fixture that is.
On the large project, the server set the floor #
Grouper is where discovery, not fixing, becomes the work. Around 300 resolved dependencies across 13 modules, no native audit command, and 56 fixable advisories spread over 21 packages that the agent first has to find.
With the server connected, no run from either agent finished below 98%. Without it, outcomes spread — Claude Code's bare runs landed at 91%, twice at 98%, and twice at 14%; Codex CLI's ranged 93–100%. If you run agents unattended, that floor is the number that matters: the catastrophic run simply stopped happening.
The 14% runs were not lazy runs. In one of them, the agent made 87 tool calls — more than any run in the study's large-project cells — and still resolved only 8 of the 56 fixable advisories. The budget went into walking the module tree by hand, not into fixing. Effort was not the problem. Knowing where to spend it was:
The fix claims also got more honest. A hallucinated fix here means the run's own report says a package was fixed, but the scored diff shows the advisory still applies — worse than a miss, because it reads as done:
Without the server, Claude Code produced at least one wrong claim in 4 of 5 runs, always on the same few awkward packages: commons-httpclient
, axis2
, the BouncyCastle pair, ion-java
. With the server, its wrong claims dropped by half — from 4 of 5 runs to 2 of 5, and from 9 affected packages to 4. For Codex CLI it was 2 of 5 in both conditions — the server did not move that number, and we report it as is.
One thing worth spelling out, because the charts raise it: a run can be 100% complete and still contain a wrong claim. Completeness counts the 56 fixable advisories. Some packages in this fixture have no working fix, and the honest outcome there is to say so. The wrong claims in the 100%-complete runs were exactly that kind: everything fixable got fixed, and the report then also claimed a "fix" for a package that cannot be fixed, instead of declining it.
And there was a speed dividend:
Codex CLI finished about 1.7× faster with the server — its slowest MCP run beat its fastest bare run. Claude Code took similar time either way and spent the saved discovery effort on remediation instead, which is where its 98% floor came from.
What we take from this #
In our setup, the Bomly MCP server mattered when discovery — not fixing — was the bottleneck. Across the ten connected runs on the large Maven fixture, every run resolved at least 98% of the fixable advisories. For Claude Code, the server eliminated the 14% runs and cut runs with wrong claims from 4 of 5 to 2 of 5. For Codex CLI, completeness was already high; the win was speed, with the average run dropping from about 13 minutes to 7.5 minutes.
One large fixture is not enough to pin down a general size threshold. What the result suggests is that whole-graph context helps once the dependency tree is hard to enumerate by hand — big, many modules, no single audit command.
The honest other half: small projects did not need the server. A good agent handles known-CVE fixes there on its own.
The study fixed four Bomly bugs #
Running our own tool underneath real agents for two weeks found four real bugs in it, which is a result we did not plan for:
- a system-installed
pip-audit
polluted Bomly's Python interpreter detection (#237) - ANSI color codes in Maven's
dependency:tree
output made every transitive Maven dependency invisible ([#243](https://github.com/bomly-dev/bomly-cli/issues/243)) - oversized MCP responses truncated half the scan calls (
[#245](https://github.com/bomly-dev/bomly-cli/issues/245)) - the Maven detector could not scan any multi-module reactor at all (
[#252](https://github.com/bomly-dev/bomly-cli/pull/252)— the large-fixture result only exists because of this fix)
FAQ #
Does the Bomly MCP server help coding agents fix vulnerable dependencies?
Yes — particularly on large projects. On a 13-module Maven project with about 300 dependencies and no native audit tool, every MCP-connected run resolved at least 98% of the fixable advisories. Without the server, Claude Code's runs ranged from 14% to 98%, while Codex CLI's stayed between 93% and 100%. On three smaller apps, capable agents did well on their own.
Does the Bomly MCP server make agents faster?
In our setup, Codex CLI was about 1.7× faster with the server on the large project. Claude Code spent similar time either way and converted the saved discovery work into more complete, more reliable remediation.
How can I reproduce the study?
Everything is in bomly-dev/bomly-agent-study: fixtures, prompts, harness, transcripts, and scoring. make verify-only
re-scores any published run with no API key.
To set this up for your own agent:
[Try Bomly MCP with your coding agent](/mcp)
Prefer a walkthrough first? Follow the step-by-step guide: [make your coding agent dependency-aware](/blog/make-your-coding-agent-dependency-aware).