ProjectDiscovery researcher Tarun Koyalwar audited open- and closed-model agents on 54 usable black-box web targets and found that most failed runs had already identified the right vulnerability but could not finish the exploit. Open models collectively covered 52 targets versus 48 for the closed group, reinforcing that solve rates alone miss execution quality, methodology, and unintended behavior.
ProjectDiscovery researcher Tarun Koyalwar published a behavioral audit of offensive-security AI agents on August 3. The corresponding BSides Las Vegas program listed the work for a Monday evening session, while the published research supplied the detailed methodology and results.
The audit followed the agents' actions
The study began with 60 black-box web targets from a patched Argus validation benchmark. Six broken targets were removed, leaving 54 usable targets. Agents received a minimal instruction to attack the application without source code or hints, and the researcher reviewed their actions instead of reducing each run to a final pass-or-fail score.
The resulting evidence separates vulnerability recognition from reliable execution. Across the admissible set, the researcher classified only one failure as a genuine discovery miss and reported that 46 runs had already targeted the right vulnerability but still failed to complete the exploit. Examples included malformed requests, repeated use of the wrong host, and unintended paths that a conventional scoreboard could count as successful even when they did not demonstrate the intended capability.
Coverage did not establish operational reliability
The open-model roster collectively reached 52 of 54 targets, compared with 48 for the closed-model group. Kimi K3 was the strongest single open model at 45 targets, with other open models adding coverage on challenges it missed. The research explicitly stops short of claiming the two groups are equal: it says the four-target difference is smaller than variation introduced by the testing harness.
Cost also varied widely. The cheapest full-corpus run cost less than one dollar, while the most expensive open-model run cost $36.64; Kimi K3's run cost $24.09. Licensing category alone therefore did not predict either coverage or operating cost in this setup.
For evaluation teams, the practical lesson is that a leaderboard should retain trajectory evidence alongside the final score. Logs that show enumeration, tool choice, recovery after errors, boundary handling, and reporting quality reveal whether an agent can perform a security workflow reliably rather than merely recognize familiar vulnerability patterns.
Key Points #
- 1The audit used 54 usable black-box web targets after discarding six broken targets from a 60-target benchmark set.
- 2Open models collectively covered 52 targets and closed models 48, but the researcher said the gap was below the measurement resolution of the harness.
- 3Most observed failures reflected execution problems after the correct vulnerability had been identified, showing why trajectory-level evidence matters alongside solve rates.
Scoring Rationale #
The research offers current, reproducible evidence about agent evaluation quality and operational failure modes, with direct relevance to security teams assessing autonomous pentesting systems.
Sources #
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.