PR Security Review Benchmark Update: New Model Showdown

GPT-5.6 Sol via OpenRouter remains the top performer in the PR security-review benchmark update, achieving 100% recall, F1 0.91, and F2 0.96 at about $0.70 per pull request. Kimi K3 is the strongest open-weights model with 88% recall and F1 0.79 but costs $0.94 per PR, while GLM 5.2 via OpenRouter is cheap at $0.13 per PR but recall-limited at 46%. Claude Opus 5 underperformed Fable on both performance and price, with F1 0.80 and $4.25 per PR.

PR Security Review Benchmark Update: New Model Showdown Summary We re-ran the same PR security-review benchmark not a full-code scan over a new wave of models: Kimi K3, Gemini 3.6, Opus 5, GLM5.2 and more OpenAI models. Overall, GPT-5.6 Sol is still on top. Kimi K3 is the strongest open-weights performer, GLM 5.2 is cheap and precise but lags hard on recall, and Opus 5 performed well but was even more expensive than Fable. More below This is a results update only. Scope, harness, scoring, and how we keep the corpus clean are unchanged. Read the original benchmark post /blog/pr-review-security-benchmark for the full methodology before treating these numbers as a general "security model" ranking. Same caveat as that post: this measures planted access-control bugs in pull requests, not freely roaming a large existing codebase. This chart plots Cost per Pull Request vs F2 performance score. Blue points define the cost/quality frontier. Highlights Headlines from the expanded run: GPT-5.6 Sol still wins. 100% perfect recall, F1 0.91 / F2 0.96, about $0.70 per PR. Nothing else we added knocked it off for reviewing the security status of PRs. Kimi K3 is the most performant open weights model , well ahead of GLM 5.2 and sitting near mid-pack closed models, but at ~$0.94 per PR it is quite expensive. GLM 5.2 via OpenRouter is cheap but recall-limited. 46% recall, 92% precision, F1 0.61. Precise when it fires, but it misses too many planted bugs for security PR review. Grok 4.5 performs well frontier at F1 0.77 but we are skeptical of the $0.20/PR price. Looks highly subsidized. Anthropic Opus 5 performed worse than Fable on both performance and price. Smart models do tend to more token efficient. GPT5.6 performed just as well as it's cheaper siblings and Fable outperforms Opus 5. Results Since the first post we added Gemini 3.6 Flash, Claude Opus 5, the rest of the GPT-5.6 stack Luna / Terra , Kimi K3, GLM 5.2, and refreshed a few earlier rows against the same harness. Same 10 planted access-control PRs, five runs each, Dam Secure Vulnerability Scanner with reasoning mode set to high on every model. Details live in the original methodology /blog/pr-review-security-benchmark methodology . | Model | Recall | Precision | F1 | F2 | Cost / PR | Cost / TP | |---|---|---|---|---|---|---| | GPT-5.6 Sol via OpenRouter | 100% | 83.3% | 0.91 | 0.96 | $0.70 | $0.70 | | GPT-5.5 via OpenRouter | 94% | 83.9% | 0.89 | 0.92 | $1.24 | $1.32 | | Gemini 3.6 Flash | 90% | 83.3% | 0.87 | 0.89 | $1.06 | $1.17 | | Fable 5 → Opus 4.8 fallback | 88% | 83% | 0.85 | 0.87 | ~$3.61 | ~$4.10 | | Claude Sonnet 4.6 | 80% | 85.1% | 0.82 | 0.81 | ~$1.22 | ~$1.53 | | GPT-5.6 Luna via OpenRouter | 90% | 73.8% | 0.81 | 0.86 | $0.76 | $0.84 | | Gemini 3.5 Flash | 84% | 76.4% | 0.80 | 0.82 | ~$0.94 | ~$1.12 | | Claude Opus 5 | 94% | 69.1% | 0.80 | 0.88 | $4.25 | $4.52 | | Kimi K3 via OpenRouter | 88% | 72.1% | 0.79 | 0.84 | $0.94 | $1.07 | | GPT-5.6 Terra via OpenRouter | 86% | 71.7% | 0.78 | 0.83 | $1.09 | $1.27 | | Grok 4.5 via OpenRouter | 74% | 80.4% | 0.77 | 0.75 | $0.20 | $0.27 | | Gemini 3.1 Flash Lite | 68% | 82.9% | 0.75 | 0.71 | ~$0.04 | ~$0.06 | | Claude Opus 4.8 | 60% | 81.1% | 0.69 | 0.63 | ~$1.72 | ~$2.87 | | Claude Haiku 4.5 | 56% | 73.7% | 0.64 | 0.59 | ~$0.75 | ~$1.34 | | GLM 5.2 via OpenRouter | 46% | 92.0% | 0.61 | 0.51 | ~$0.13 | ~$0.28 | | DeepSeek V4 Pro via OpenRouter | 30% | 65.2% | 0.41 | 0.34 | $0.21 | $0.71 | True Positive TP - a vuln found that was expected to be found. False Positive FP - a vuln found that was not expected not a real bug . False Negative FN - a vuln not found that was expected to be found. Recall R - how well did it find the planted vulnerabilities? It is the fraction of actual positives correctly identified: TP / TP + FN . Precision P - false positive performance. A fraction of predicted positives that are correct: TP / TP + FP . F1 - balances precision and recall; use it when reducing false-positive noise for developers matters: 2PR / P + R . F2 - favors recall; use it when finding more true positives matters most: 5PR / 4P + R . Costs / PR - is the cost to run that model over all prompts and process the PR regardless of it's accuracy. This is the most accurate representation of real-world cost. Cost / TP - is the cost per true positive found.- Wikipedia: Precision, Recall and F1 https://en.wikipedia.org/wiki/Precision and recall More notes Again: PR review only. Full methodology is in the original article /blog/pr-review-security-benchmark .