# Benchmarking 13 AI models on rediscovering known CVEs > Source: > Published: 2026-07-16 11:24:00+00:00 TL;DR **GPT-5.6 gets the highest recall score,** topping out at 23/26, ahead of grok-4.5 (20), the Claude Opus models (15 to 18), and everything else we tested. That means**it can rediscover 88.5% of CVEs.****The most expensive option is not required.** GPT-5.6's cheaper variants land within one or two findings of its flagship at a fraction of the cost, and pooling a few runs of a mid-tier model rivals a single pass of a flagship.**Model runs are inconsistent, but pooling means they win.** Any single run misses bugs it would catch on another; running a model a few times and pooling the results (pass@3) reliably beats one pass of a stronger, pricier model.**Open weight is catching up fast.** GLM-5.2 already rediscovers 59% of the set (16/26), mid-pack among the proprietary models. Every frontier model launch now comes with the same cybersecurity claim: *it finds vulnerabilities*. But does it work on a real bug in a real repository, or just on a curated example? Of the dozen models you could pick, which is worth trusting with code review? And since the strongest models cost ten times or more per run than the cheapest, what does that extra spend actually buy you in bugs found? It is easy to rank models by raw capability and call the priciest one the winner, but the more important question is if the price is worth it. So we ran 13 of the models teams are choosing between today against 26 known vulnerabilities from the GitHub advisory database. They span a range of languages and project types. We measured two things: how many bugs each model found, and what it cost to find them. ## How the benchmark works We took 26 vulnerabilities from the GitHub advisory database, a random spread across languages and project types, from SQL injection in a web framework to a deserialization RCE in an ML toolkit, and asked each model to rediscover them one repository at a time inside the same [AI Code Analysis](https://www.aikido.dev/code/code-audit) harness we run in production. Rather than a chat window, it’s a model with real tools that navigates the repo and reasons about code the way an auditor does. The harness is what turns a language model into an auditor. A general-purpose coding assistant is built for a different job, which is to take a task and produce working code. Point it at a repository and ask whether it is secure, and it behaves like a developer skimming for something obviously broken, then stops once it has found something plausible. [AI Code Analysis](https://www.aikido.dev/code/code-audit) is built differently. It scouts the codebase for candidate entry points, investigates each suspicious flow in depth, then triages what comes back so only real vulnerabilities survive. Because we knew where each vulnerability lived, we pointed every investigator agent straight at the vulnerable code snippet. That way a miss reflects reasoning rather than budget wasted wandering the wrong corner of the codebase. The model still has to understand the flow, judge exploitability, and report it correctly. Prompts were kept short and model-agnostic so no vendor is advantaged by wording. We ran each model three times and pooled the results. A CVE is considered ‘found’ if the model surfaces it on any run (pass@3). We chose a variety of the latest models from different vendors: **OpenAI**: gpt-5.4-nano, gpt-5.4-mini, gpt-5.5, and the gpt-5.6 series (luna / terra / sol)** Anthropic:**claude-haiku-4-5, claude-opus-4-7, claude-opus-4-8** xAI:**grok-4.5** Google:**gemini-3.1-pro, gemini-3.5-flash** Open weight:**glm-5.2 ## Results by severity and vulnerability Every model rediscovered both critical CVEs (a deserialization RCE and a stored XSS). The real separation shows up on the high- and medium-severity findings. ### Hardest & easiest CVEs The two critical bugs and several clear injection/access-control flaws were found by every model. A handful of specific chains defeated nearly all of them. The CVEs every model finds share the same pattern: attacker-controlled inputs reaching a well-known dangerous operation over a short, local flow, such as a deserialization call, a shell exec, an HTML sink, or a broken signature check. This is pattern recognition, and it is effectively solved. Cheap models and flagships alike score them 13/13, with no capability separation at all. The real frontier, and where model capability actually separates, is reasoning about the checks that are not there and following complex chains no single line gives away. The clearest case is *SQL Injection-1* in our dataset, an indirect injection through a column alias the ORM never escapes. Only GPT-5.5 and the strongest GPT-5.6 models (sol & terra) traced it. ## What the difference between average and union tells us The most useful number in this benchmark is the distance between a model's average run and the union of its runs. Because each pass surfaces a different subset of the bugs, pooling them (pass@3) recovers a surprising amount: Look at gpt-5.4-nano: no single run cracks 14, yet three pooled runs reach 18, a four-CVE jump, because each pass surfaces different bugs. claude-haiku-4-5 is the starkest case of variance, scoring 7 on one run and 13 on another from the same model on the same task. The takeaway here is that you do not have to reach straight for the most expensive model. Three runs of gpt-5.4-nano cost about $170 and reach 18/26, as much as a single pass of a flagship like gpt-5.6-terra averages on its own, for a fraction of the price. Three runs of gpt-5.4-mini (about $460) reach 20. Repeating a solid mid-tier model beats a single pass of a flagship far more often than the price difference suggests. ## Is higher reasoning worth it? We ran the applicable models at two reasoning levels, the default "high" and their highest available ("xhigh" for the GPT-5.4/5.5 and Claude models, "max" for GPT-5.6, grok-4.5, and glm-5.2). All figures are pass@3 unions, so they are directly comparable. The clear wins are gpt-5.5 (+3 at 1.5x the cost) and glm-5.2 (+3 at 1.3x). claude-opus-4-8 gains +2 but pays double for it. Everywhere else, the top tier buys one finding or none, and for gpt-5.6-luna and gpt-5.4-nano it landed one lower, within run-to-run noise. gpt-5.6-terra is the sharpest case of diminishing returns: 2.2x the cost for the same 23. The Gemini models have no setting above "high," so they are omitted from this comparison. ## Conclusion Point a frontier model at a codebase, inside a harness built for the job, and it rediscovers most known vulnerabilities. Bugs with an obvious dangerous sink are solved. The ones that still separate models have no sink to point at. A missing authorization check, or an injection you can only reach by following a long obscure chain across several files. The most expensive tier rarely earns its price. Running a cheaper model a few times and pooling the results finds more, for less, than a single pass of a flagship, and that edge only grows as models get cheaper and stronger. The harness is still what decides whether that reasoning gets aimed at the right part of the codebase in the first place.