Summary #
Two weeks ago we planted vulnerabilities in 16 repos and ran the latest models over them. Since then GPT-6 Sol, GPT-6 Luna, Claude Opus 5.5 and Grok 4.7 have shipped, so we ran them through the same 16 pull requests. DeepSeek 4.1 Flash is still the best value: 64% of the planted bugs found for $0.42 a PR. GPT-6 Sol now matches that recall in about two minutes a PR, for $1.81. It is circled on each chart.
This time we also looked at which bugs each model finds, how much that changes from run to run, and how many tokens it takes.
Highlights #
- GPT-6 Sol is closing in on DeepSeek v4.1 Flash. Same 64% recall, in 2m 17s per PR instead of 6m 32s, for $1.81 instead of $0.42.
- The hard bugs decide the top of the table. Nine of the 16 bugs are found by almost every capable model. Four are found by almost nobody. Grok 4.7 has the highest recall because it finds some of those four, not because it is better at the rest.
- Some models are consistent, others are lucky. GPT-6 Astra and Claude Opus 5 find the same bugs on every run. DeepSeek v4.1 Flash and GPT-6 Sol vary, so two runs of either find 71% of the bugs, more than one run of Astra, at a fraction of the cost.
- Thinking harder doesn't mean paying more. DeepSeek v4.1 Flash produces 15 times the reasoning tokens of GPT-6 Sol and still costs a quarter as much per PR.
- Claude Opus 5.5 is the fastest review we have measured : 1m 26s per PR. It thinks about a ninth as much as Opus 5, and recall dropped from 64% to 56%.
- GPT-6 Luna is nearly free, and misses most bugs. $0.07 per PR, 40% recall.
Results #
| Model | Recall | Cost / PR | Time / PR | Findings / TP | Reasoning tokens / PR |
|---|---|---|---|---|---|
| Grok 4.7 (via Vercel) | 77.5% | $9.28 | 18m 23s | 5.8 | 282,099 |
| GPT-6 Astra | 67.5% | $8.80 | 5m 13s | 6.6 | 14,687 |
| DeepSeek V4.1 Flash (via Vercel) | 63.75% | $0.42 | 6m 32s | 8.8 | 336,871 |
| GPT-6 Sol | 63.75% | $1.81 | 2m 17s | 7.9 | 22,462 |
| Claude Opus 5 | 63.75% | $11.08 | 7m 10s | 7.8 | 118,007 |
| Fable 5.1 | 62.5% | $7.31 | 3m 44s | 6.0 | 42,924 |
| GPT-5.6 Sol | 61.25% | $5.82 | 3m 33s | 8.4 | 36,013 |
| GLM 5.3 (via OpenRouter) | 57.5% | $2.09 | 6m 43s | 8.4 | 104,084 |
| Claude Opus 5.5 | 56.25% | $1.92 | 1m 26s | 4.9 | 13,701 |
| Kimi K3 (via OpenRouter) | 56.25% | $4.01 | 8m 28s | 9.0 | 70,728 |
| GLM 5.3 Flash (via OpenRouter) | 48.75% | $0.11 | 8m 6s | 10.3 | 99,728 |
| Grok 4.6 (via OpenRouter) | 47.5% | $4.66 | 4m 42s | 5.1 | 105,881 |
| GPT-6 Luna | 40% | $0.07 | 2m 1s | 7.5 | 39,569 |
| GPT-5.6 Luna | 38.75% | $0.28 | 2m 52s | 10.6 | 42,329 |
| Gemini 3.8 Flash | 35% | $3.45 | 4m 32s | 5.1 | 236,154 |
| Claude Sonnet 5 | 32.5% | $3.18 | 5m 23s | 14.4 | 88,941 |
Same definitions as last time, with three clarifications:
- Findings / TP is total findings divided by planted bugs found, for every model. Some rows in the previous post divided by reviews instead, so numbers for the same model differ between the two posts.
- Reasoning tokens / PR is the average number of reasoning tokens a model produced per review. The previous post showed token totals summed over all 80 reviews.
- For models served through OpenRouter, Time / PR is normalised from 50 to 80 tokens per second.
Mean review time per PR vs recall. Blue points define the time/quality frontier.
Under 10 minutes, GPT-6 Astra still finds the most (68%). GPT-6 Sol and Claude Opus 5.5 are the fastest reviews that still catch more than half the planted bugs. Grok 4.7 finds the most (78%), but at 18 minutes a PR it is well past our 10-minute budget.
Findings per true positive vs recall. Lower on this axis means less junk for developers to triage. Blue points define the noise/quality frontier.
Claude Opus 5.5 is the quietest model that catches more than half the bugs: 4.9 findings per true positive. DeepSeek v4.1 Flash is still one of the noisiest leaders at 8.8; GPT-6 Sol is a little quieter at 7.9.
Easy bugs, hard bugs #
Across all 16 models and 80 reviews per bug, the 16 planted bugs fall into three clear groups:
| Group | Bugs | Found in |
|---|---|---|
| Easy | 9 | 70–86% of reviews |
| Medium | 3 | 36–56% of reviews |
| Hard | 4 | 0–9% of reviews |
The easy group is where most models look alike. Claude Opus 5 found every easy bug on every run, and DeepSeek v4.1 Flash, Fable 5.1 and Grok 4.7 missed one easy review in 45.
The hard group is where the leaders separate:
| Hard bug | Found by |
|---|---|
| Server-side request forgery (SSRF) | None of the 16 |
| Stale session accepted after it should be invalid | Grok 4.7 (5 of 5 runs), GPT-6 Sol (1 of 5), Claude Sonnet 5 (1 of 5) |
| Stale authorization decision reused | GPT-5.6 Sol (3 of 5 runs), Grok 4.7 (1 of 5), Claude Sonnet 5 (1 of 5) |
| Identity check that proves the wrong thing | DeepSeek v4.1 Flash (1 of 5 runs), Fable 5.1 (1 of 5), Kimi K3 (1 of 5) |
Three of the four hard bugs are authentication or authorization logic that depends on state the diff doesn't show: a session, an earlier decision, or who the caller really is. Grok 4.7 caught a hard bug in 6 of its 20 attempts. The next best, GPT-5.6 Sol, caught 3. GPT-6 Astra and Claude Opus 5 found none.
Nobody found the SSRF in 80 attempts. A benchmark with no unsolved bugs can't tell the next generation of models apart, so we keep it.
Run it twice? #
Every model reviewed every PR five times, so we can see how much each one varies:
| Model | Found every run | Found sometimes | Never found | 1 run | 2 runs | 3 runs |
|---|---|---|---|---|---|---|
| Grok 4.7 | 10 | 4 | 2 | 78% | 83% | 85% |
| GPT-6 Astra | 10 | 1 | 5 | 68% | 69% | 69% |
| Claude Opus 5 | 10 | 1 | 5 | 64% | 65% | 66% |
| DeepSeek v4.1 Flash | 8 | 5 | 3 | 64% | 71% | 76% |
| GPT-6 Sol | 8 | 5 | 3 | 64% | 71% | 76% |
| Fable 5.1 | 8 | 5 | 3 | 63% | 69% | 74% |
| Claude Opus 5.5 | 7 | 3 | 6 | 56% | 61% | 62% |
Bug counts are out of 16. The 2- and 3-run columns are the expected recall if you combine that many independent runs, estimated from the five we have.
GPT-6 Astra and Claude Opus 5 are almost deterministic: they find the same 10 bugs every time and never find 5 of them. Running them again buys nothing.
DeepSeek v4.1 Flash and GPT-6 Sol are the opposite. Five bugs show up on some runs and not others, so a second run lifts recall from 64% to 71%. That beats one run of GPT-6 Astra (68%) for $0.85 (DeepSeek) or $3.62 (Sol), against $8.80. Runs can go in parallel, so the review takes no longer. The catch is noise: two runs means two sets of findings to merge and deduplicate before a developer sees them.
Across a single run, Grok 4.7 found between 10 and 13 of the 16 bugs. Claude Sonnet 5 ranged from 2 to 8. A single benchmark run can move a model several places, which is why we run five.
Tokens aren't the bill #
| Model | Reasoning tokens / PR | Recall | Cost / PR | Cost / bug found |
|---|---|---|---|---|
| Claude Opus 5.5 | 14k | 56% | $1.92 | $3.41 |
| GPT-6 Astra | 15k | 68% | $8.80 | $13.03 |
| GPT-6 Sol | 22k | 64% | $1.81 | $2.84 |
| Claude Opus 5 | 118k | 64% | $11.08 | $17.38 |
| Gemini 3.8 Flash | 236k | 35% | $3.45 | $9.84 |
| Grok 4.7 | 282k | 78% | $9.28 | $11.98 |
| DeepSeek v4.1 Flash | 337k | 64% | $0.42 | $0.67 |
The OpenAI models and Claude Opus 5.5 think in the tens of thousands of tokens per PR. DeepSeek, Grok and Gemini think in the hundreds of thousands. Token count alone predicts neither recall nor cost. DeepSeek v4.1 Flash uses 15 times GPT-6 Sol's reasoning tokens for the same recall and is still the cheapest per bug found, because its tokens are so cheap. Gemini 3.8 Flash thinks nearly as much as Grok 4.7 and finds less than half as many bugs.
More thinking does help on the hard bugs. Grok 4.7, which found the most of them, is one of the heaviest thinkers and the slowest model we tested. Claude Opus 5.5 went the other way: about a ninth of Opus 5's reasoning, five times faster, a sixth of the cost, and recall down from 64% to 56%.
What we're doing with this #
- Day-to-day PR review: DeepSeek v4.1 Flash on cost, GPT-6 Sol if the review needs to land in about two minutes.
- Consistency matters as much as recall. A model that finds a bug one run in five will miss it in production four times out of five. For the models that vary, a second run in parallel is cheaper than a pricier model.
- One model is still not enough. No single model found all of the hard bugs, and the ones that found the most took too long. Dam Secure routes reviews to the models that perform best for each workload and keeps measuring as new models arrive.Get started .
Methodology #
Same benchmark, harness settings and scoring as the September 10 post: 16 pull requests with a reverse-replayed vulnerability each, reasoning effort set to high, five runs per model (80 scored reviews). Results for the models in the previous post come from the same runs. The four new models were scored by three independent blind grading passes, then cross-checked by a fourth with a different model; all four agreed on every review.