{"slug": "deepsecbench-evaluating-model-performance-in-finding-cybersecurity", "title": "DeepsecBench: evaluating model performance in finding cybersecurity vulnerabilities", "summary": "OpenAI evaluated two models on an exploit benchmark within an isolated sandbox, where the models found a vulnerability, accessed the internet, and reached Hugging Face's production database without human direction. Today, Vercel released DeepsecBench, a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code, with the best run finding 30.7% of known vulnerabilities. The benchmark uses an open-source codebase at a commit state just before vulnerabilities were fixed, with 50 entry-point files and 231 human-judged findings, and scores are recall-weighted F2.", "body_md": "Last week, OpenAI evaluated two models on an exploit benchmark within an isolated sandbox. Guardrails were reduced for testing, and the models found a vulnerability in their environment, accessed the internet, and reached Hugging Face's production database.\n\nNo human directed the action, but the breach is a clear example of how much more capable malicious attackers are when equipped with powerful AI models. But defenders have the same tools, and a clear advantage: knowledge of their own codebase. Hacks are initiated from the outside, so the single best defense is finding vulnerabilities from the inside before attackers do.\n\nToday we're releasing [DeepsecBench](https://vercel.com/ai-gateway/leaderboards/deepsecbench), a benchmark that evaluates how well different models find cybersecurity vulnerabilities in application code. For each model the report includes recall, precision, cost, and total time, and combines recall and precision into a single benchmark score. Here is a sample of model performance from the leaderboard:\n\nRank | Model | Level | Score | Cost | Total time |\n|---|---|---|---|---|---|\n1 |\n| xhigh |\n| $55.98 | 03:39:00 |\n3 |\n| medium |\n| $31.96 | 00:47:01 |\n8 |\n| high |\n| $12.38 | 01:59:00 |\n10 |\n| high |\n| $5.60 | 01:24:00 |\n\nWe built [deepsec](https://deepsec.sh/) to make scanning as easy as possible. Now you can use the benchmark report to build a security scanning program that fits your budget and the complexity of your codebase, choosing the right mix of models to run and how often to run them.\n\nDeepsecBench runs on an open-source codebase at a commit state just before a large number of vulnerabilities were fixed. We selected 50 entry-point files and built a golden set of 231 human-judged findings. Each model's score is a recall-weighted F2 (Score = 100 × 5PR/(4P+R)), weighting recall (R) twice as much as precision (P), because missed vulnerabilities will go unfixed, while false positives don't make your codebase less secure.\n\nFindings beyond the golden set are classified by a judge model as real or false, and count for or against the model in the precision measure (recall measures only the 231 known findings).\n\nThe benchmark is run three times and the data published in the report is the median of the three runs.\n\nThe construction of the benchmark stays secret. We don't disclose the repository, the commit, the files, or the findings, so there is nothing for models to train against. A model reciting memorized fixes would score near-total recall. Instead, the best run finds 30.7%, and 20 of the 25 runs come in under 20%.\n\nWhen we [introduced deepsec](https://vercel.com/blog/introducing-deepsec-find-and-fix-vulnerabilities-in-your-code-base), thorough scanning on production codebases required using the most capable models, and they were expensive to run. Frontier models from OpenAI and Anthropic still score highest, but open-weight models and more efficient reasoning options are closing that gap, making comprehensive scanning far more cost-efficient.\n\nToday, higher price does not buy proportionally more. [Kimi K3](https://vercel.com/ai-gateway/models/kimi-k3), from Moonshot AI, ranks eighth at a score of 17.56 for $12.38 on the high setting, half the top score for about a fifth of the cost. [Grok 4.5](https://vercel.com/ai-gateway/models/grok-4.5) set to high delivers near-Kimi performance for less than half the cost, scoring 15.58 for only $5.60. [GPT-5.6 Sol](https://vercel.com/ai-gateway/models/gpt-5.6-sol?__vercel_draft=1) set to medium offers the best score-to-cost balance of the top-performing models, taking fifth place with a score of 25.10 at a cost of $17.95.\n\nAnthropic's most capable model, Fable 5, is absent because it declines security work, including defensive tasks. We will add security-enabled versions to the benchmark when they are made available.\n\nRank | Model | Level | Score | Recall | Precision | Issues | Cost | Total time |\n|---|---|---|---|---|---|---|---|---|\n1 |\n| xhigh |\n| 30.7% | 96.3% | 71/231 | $55.98 | 03:39:00 |\n2 |\n| max |\n| 28.1% | 88.0% | 65/231 | $127.93 | 02:34:00 |\n3 |\n| medium |\n| 24.7% | 70.5% | 57/231 | $31.96 | 00:47:01 |\n4 |\n| xhigh |\n| 22.9% | 81.2% | 53/231 | $24.59 | 03:01:00 |\n5 |\n| medium |\n| 21.2% | 94.3% | 49/231 | $17.95 | 00:30:52 |\n6 |\n| xhigh |\n| 17.7% | 95.5% | 41/231 | $43.42 | 02:22:00 |\n7 |\n| xhigh |\n| 16.0% | 95.3% | 37/231 | $27.81 | 01:45:00 |\n8 |\n| high |\n| 14.7% | 77.1% | 34/231 | $12.38 | 01:59:00 |\n9 |\n| medium |\n| 13.9% | 73.3% | 32/231 | $11.04 | 01:37:00 |\n10 |\n| high |\n| 13.0% | 77.8% | 30/231 | $5.60 | 01:24:00 |\n11 |\n| medium |\n| 10.4% | 92.3% | 24/231 | $14.94 | 00:24:05 |\n12 |\n| medium |\n| 10.0% | 92.0% | 23/231 | $5.15 | 00:13:15 |\n13 |\n| medium |\n| 10.0% | 82.8% | 23/231 | $5.23 | 00:11:59 |\n14 |\n| medium |\n| 9.5% | 80.0% | 22/231 | $10.74 | 00:26:21 |\n15 |\n| medium |\n| 9.5% | 64.1% | 22/231 | $10.84 | 01:29:00 |\n16 |\n| medium |\n| 8.7% | 62.5% | 20/231 | $11.09 | 01:03:00 |\n17 |\n| high |\n| 8.2% | 53.8% | 19/231 | $51.84 | 02:30:00 |\n18 |\n| high |\n| 7.4% | 77.3% | 17/231 | $5.33 | 00:22:18 |\n19 |\n| max |\n| 5.6% | 81.3% | 13/231 | $7.56 | 00:19:29 |\n20 |\n| medium |\n| 5.6% | 61.9% | 13/231 | $10.81 | 00:18:16 |\n21 |\n| high |\n| 5.2% | 48.0% | 12/231 | $1.93 | 00:11:34 |\n22 |\n| medium |\n| 4.3% | 37.0% | 10/231 | $2.18 | 00:10:31 |\n23 |\n| default |\n| 3.9% | 33.3% | 9/231 | $5.91 | 00:29:40 |\n24 |\n| medium |\n| 1.7% | 44.4% | 4/231 | $0.58 | 00:01:46 |\n25 |\n| high |\n| 0.9% | 28.6% | 2/231 | $0.18 | 00:01:07 |\n\nThe costs reported in the benchmark cover 50 files. A production codebase can run on the order of 100 times that, pricing a full pass at roughly $1,200 for a Kimi K3 sweep, or over $5,000 for the top-scoring frontier model from OpenAI.\n\nBecause security scanning is recurring work, the question is which model to use for which task, and when. For example, frontier models can be used for periodic deep audits, while cheaper models like Kimi K3 or Grok 4.5 run scans at a higher cadence. Turning GPT-5.6 Sol's reasoning down from xhigh to medium moves it from 35.58 in 3 hours, 39 minutes to 25.10 in just over 30 minutes, fast enough for reviewing new features before they are pushed to production.\n\nA startup might scan every merge with Grok 4.5 and save more expensive audits for milestone releases. A large enterprise might run frontier audits on its critical services, the same model at lower reasoning on key pull requests, and a continuous Kimi-class or Grok sweep across the rest of the codebase.\n\nSecurity scans are spiky workloads that consume a large number of tokens in a short time window, and that burst capacity is hard to buy from any single provider directly. Every DeepsecBench run goes through [AI Gateway](https://vercel.com/ai-gateway). The gateway gives us a single endpoint to reach each model. During runs, routing, retries, and failover happen automatically, without any per-provider keys or rate limits to manage.\n\nThe same routing we use for the benchmark can run your security scanning program. Pass `provider/model`\n\nto `deepsec`\n\n, and one `AI_GATEWAY_API_KEY`\n\nin your environment covers every model on the board (or on a linked Vercel project, access AI Gateway via OIDC).\n\nAttackers can use AI models, but they act from the outside. Defenders can see their entire system: the source, the architecture, the history. That visibility makes the same models more powerful in your hands than in theirs, because a model that can read the source finds what an attacker can only probe for. The advantage is real, but it only counts if you use it first.\n\nWe will continue to update [DeepsecBench](https://vercel.com/ai-gateway/leaderboards/deepsecbench) with new models as they are released and measured.", "url": "https://wpnews.pro/news/deepsecbench-evaluating-model-performance-in-finding-cybersecurity", "canonical_source": "https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-vulnerabilities", "published_at": "2026-07-27 04:00:00+00:00", "updated_at": "2026-07-27 20:07:48.620764+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-safety", "ai-research"], "entities": ["OpenAI", "Hugging Face", "Vercel", "DeepsecBench", "Moonshot AI", "Kimi K3", "Grok 4.5", "GPT-5.6 Sol"], "alternates": {"html": "https://wpnews.pro/news/deepsecbench-evaluating-model-performance-in-finding-cybersecurity", "markdown": "https://wpnews.pro/news/deepsecbench-evaluating-model-performance-in-finding-cybersecurity.md", "text": "https://wpnews.pro/news/deepsecbench-evaluating-model-performance-in-finding-cybersecurity.txt", "jsonld": "https://wpnews.pro/news/deepsecbench-evaluating-model-performance-in-finding-cybersecurity.jsonld"}}