{"slug": "pr-security-review-benchmark-update-new-model-showdown", "title": "PR Security Review Benchmark Update: New Model Showdown", "summary": "GPT-5.6 Sol via OpenRouter remains the top performer in the PR security-review benchmark update, achieving 100% recall, F1 0.91, and F2 0.96 at about $0.70 per pull request. Kimi K3 is the strongest open-weights model with 88% recall and F1 0.79 but costs $0.94 per PR, while GLM 5.2 via OpenRouter is cheap at $0.13 per PR but recall-limited at 46%. Claude Opus 5 underperformed Fable on both performance and price, with F1 0.80 and $4.25 per PR.", "body_md": "# PR Security Review Benchmark Update: New Model Showdown\n\n## Summary\n\nWe re-ran the same **PR security-review** benchmark (not a full-code scan) over a new wave of models: Kimi K3, Gemini 3.6, Opus 5, GLM5.2 and more OpenAI models. Overall, GPT-5.6 Sol is still on top. Kimi K3 is the strongest open-weights performer, GLM 5.2 is cheap and precise but lags hard on recall, and Opus 5 performed well but was even more expensive than Fable. More below!\n\nThis is a results update only. Scope, harness, scoring, and how we keep the corpus clean are unchanged. Read the [original benchmark post](/blog/pr-review-security-benchmark) for the full methodology before treating these numbers as a general \"security model\" ranking. Same caveat as that post: this measures planted access-control bugs in pull requests, not freely roaming a large existing codebase.\n\n*This chart plots Cost per Pull Request vs F2 performance score. Blue points define the cost/quality frontier.*\n\n## Highlights\n\nHeadlines from the expanded run:\n\n**GPT-5.6 Sol still wins.** 100% (perfect) recall, F1 0.91 / F2 0.96, about $0.70 per PR. Nothing else we added knocked it off for reviewing the security status of PRs.**Kimi K3 is the most performant open weights model**, well ahead of GLM 5.2 and sitting near mid-pack closed models, but at ~$0.94 per PR it is quite expensive.** GLM 5.2 via OpenRouter is cheap but recall-limited.**46% recall, 92% precision, F1 0.61. Precise when it fires, but it misses too many planted bugs for security PR review.** Grok 4.5 performs well**(frontier at F1 0.77) but we are skeptical of the $0.20/PR price. Looks highly subsidized.** Anthropic**Opus 5 performed worse than Fable on both performance and price.** Smart models do tend to more token efficient.**GPT5.6 performed just as well as it's cheaper siblings and Fable outperforms Opus 5.\n\n## Results\n\nSince the first post we added Gemini 3.6 Flash, Claude Opus 5, the rest of the GPT-5.6 stack (Luna / Terra), Kimi K3, GLM 5.2, and refreshed a few earlier rows against the same harness. Same 10 planted access-control PRs, five runs each, Dam Secure Vulnerability Scanner with reasoning mode set to high on every model. Details live in the [original methodology](/blog/pr-review-security-benchmark#methodology).\n\n| Model | Recall | Precision | F1 | F2 | Cost / PR | Cost / TP |\n|---|---|---|---|---|---|---|\n| GPT-5.6 Sol via OpenRouter | 100% | 83.3% | 0.91 | 0.96 | $0.70 | $0.70 |\n| GPT-5.5 via OpenRouter | 94% | 83.9% | 0.89 | 0.92 | $1.24 | $1.32 |\n| Gemini 3.6 Flash | 90% | 83.3% | 0.87 | 0.89 | $1.06 | $1.17 |\n| Fable 5 → Opus 4.8 fallback | 88% | 83% | 0.85 | 0.87 | ~$3.61 | ~$4.10 |\n| Claude Sonnet 4.6 | 80% | 85.1% | 0.82 | 0.81 | ~$1.22 | ~$1.53 |\n| GPT-5.6 Luna via OpenRouter | 90% | 73.8% | 0.81 | 0.86 | $0.76 | $0.84 |\n| Gemini 3.5 Flash | 84% | 76.4% | 0.80 | 0.82 | ~$0.94 | ~$1.12 |\n| Claude Opus 5 | 94% | 69.1% | 0.80 | 0.88 | $4.25 | $4.52 |\n| Kimi K3 via OpenRouter | 88% | 72.1% | 0.79 | 0.84 | $0.94 | $1.07 |\n| GPT-5.6 Terra via OpenRouter | 86% | 71.7% | 0.78 | 0.83 | $1.09 | $1.27 |\n| Grok 4.5 via OpenRouter | 74% | 80.4% | 0.77 | 0.75 | $0.20 | $0.27 |\n| Gemini 3.1 Flash Lite | 68% | 82.9% | 0.75 | 0.71 | ~$0.04 | ~$0.06 |\n| Claude Opus 4.8 | 60% | 81.1% | 0.69 | 0.63 | ~$1.72 | ~$2.87 |\n| Claude Haiku 4.5 | 56% | 73.7% | 0.64 | 0.59 | ~$0.75 | ~$1.34 |\n| GLM 5.2 via OpenRouter | 46% | 92.0% | 0.61 | 0.51 | ~$0.13 | ~$0.28 |\n| DeepSeek V4 Pro via OpenRouter | 30% | 65.2% | 0.41 | 0.34 | $0.21 | $0.71 |\n\n*True Positive (TP)*- a vuln found that was expected to be found.*False Positive (FP)*- a vuln found that was not expected (not a real bug).*False Negative (FN)*- a vuln not found that was expected to be found.*Recall (R)*- how well did it find the planted vulnerabilities? It is the fraction of actual positives correctly identified: TP / (TP + FN).*Precision (P)*- false positive performance. A fraction of predicted positives that are correct: TP / (TP + FP).*F1*- balances precision and recall; use it when reducing false-positive noise for developers matters: 2PR / (P + R).*F2*- favors recall; use it when finding more true positives matters most: 5PR / (4P + R).*Costs / PR*- is the cost to run that model over all prompts and process the PR regardless of it's accuracy. This is the most accurate representation of real-world cost.*Cost / TP*- is the cost per true positive found.- Wikipedia:\n[Precision, Recall and F1](https://en.wikipedia.org/wiki/Precision_and_recall)\n\n## More notes\n\nAgain: PR review only. Full methodology is in the [original article](/blog/pr-review-security-benchmark).", "url": "https://wpnews.pro/news/pr-security-review-benchmark-update-new-model-showdown", "canonical_source": "https://docs.damsecure.ai/blog/pr-review-security-benchmark-update", "published_at": "2026-07-30 00:00:00+00:00", "updated_at": "2026-08-09 13:00:06.095671+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-research", "ai-products"], "entities": ["OpenRouter", "GPT-5.6 Sol", "Kimi K3", "GLM 5.2", "Claude Opus 5", "Gemini 3.6 Flash", "Grok 4.5", "Fable"], "alternates": {"html": "https://wpnews.pro/news/pr-security-review-benchmark-update-new-model-showdown", "markdown": "https://wpnews.pro/news/pr-security-review-benchmark-update-new-model-showdown.md", "text": "https://wpnews.pro/news/pr-security-review-benchmark-update-new-model-showdown.txt", "jsonld": "https://wpnews.pro/news/pr-security-review-benchmark-update-new-model-showdown.jsonld"}}