Collinear AI has opened a public version of CWE-bench, a test that asks coding agents to find and fix real security weaknesses. According to its Hugging Face post dated November 3, 2026, CWE-bench v2 dataset contains 100 public tasks across 91 different CWE types and eight programming languages. Among 11 models evaluated, Gemini 4 Argon leads with 67.8% pass@1, just ahead of Claude Opus 5.5 at 67.3%.
These scores are Collinear's, from its own harness. The tasks, grader and harness are open, so anyone can rerun them, which is the main reason the release matters. A benchmark you can reproduce beats a leaderboard you must trust.
TL;DR: CWE-bench v2 in a table #
| Question | Answer |
|---|---|
| What is it? | 100 defensive audit-and-patch tasks, public |
| Coverage | 91 CWE types, 8 languages |
| Task | Find and fix one weakness without breaking existing behavior |
| Grading | Deterministic verifier; reward 1 only if all exploits closed and all legitimate cases pass |
| Top score | Gemini 4 Argon, 67.8% pass@1, 81.0% pass@4 |
| Second | Claude Opus 5.5, 67.3% pass@1, 78.0% pass@4 |
| Cheapest strong result | DeepSeek V4.1 Flash (1017): 57.0% pass@1 at $0.04 per rollout |
| Tool | Harbor; gated dataset on Hugging Face |
| Held-out set | Kept private; contact cyber@collinear.ai |
Agents that act on your behalf need rules. See how explainx helps secure your agents →
What an agent does in each task #
Per Collinear, the agent receives a real open-source repository and a short instruction. The goal is to find and fix a security weakness without breaking existing functionality. The dataset card says each task contains a codebase with one weakness, and a fix is accepted only when the weakness is closed and legitimate behavior is preserved.
The language split on the dataset page: Go 27 tasks, Java 15, JavaScript 15, C++ 14, C 10, Python 10, Rust 5, TypeScript 4. Each task is a self-describing Harbor task with metadata, an instruction file, a Dockerfile that builds the vulnerable code with dependencies for offline use, a reference fix, and a test script that runs the deterministic verifier.
Reward is binary. It is 1 only when the verifier finds every exploit input closed and every legitimate case working. If the environment fails to build, the trial is void, not scored zero.
How Collinear says it kept the tasks fair #
The post lists six checks applied to every task:
- Does the instruction read like a realistic instruction rather than an evaluation setting?
- Can every graded behavior be inferred from the instruction?
- Does the verifier check correct behavior, not the structure of one fix?
- Does the starter code fail and the reference fix pass?
- Does the original functionality still work with the new fix?
- Do the verifiers fail memorized solutions such as the known upstream patch?
Check six is the interesting one. A model trained on public code may have seen the real patch. Requiring that verifiers reject the memorized upstream fix pushes the test toward understanding. LLM judge panels helped curate the programmatic checks toward hardness and fairness, and every task was reviewed by hand. Collinear also keeps a held-out set so it can track progress on tasks no model has seen.
The results #
Collinear reports two places on the Hub. The dataset page leaderboard covers open-weight models and updates when labs or the community open a PR on a model repo with a results file. A separate leaderboard Space covers all models, including API-only ones such as Gemini, Claude and GPT.
| Model | pass@1 | pass@4 | Cost note |
|---|---|---|---|
| Gemini 4 Argon | 67.8% | 81.0% | $3.92 per rollout |
| Claude Opus 5.5 | 67.3% | 78.0% | $1.83 per rollout |
| DeepSeek V4.1 Flash (1017) | 57.0% | not stated | $0.04 per rollout |
Open-weight scores from the dataset page, as shown when we looked: DeepSeek V4.1 Flash 1017 at 57, Tencent Hy4-preview at 52, Mistral Large 4 at 46.3 and GLM-5.3 at 44. The page marks these with an asterisk; check its notes before comparing them with the all-model table, since the setups may differ.
Three takeaways from Collinear's numbers:
- Solvable but not solved. 95 of 100 tasks are solved by at least one model, so the tasks are possible. Yet even the leader fails 19% of the set on every attempt.
- Pass@4 matters. Gemini 4 Argon rises from 67.8% to 81.0% with four tries. If you can verify a fix, retrying is cheap insurance.
- Cost frontier. Collinear says DeepSeek V4.1 Flash, Claude Opus 5.5 and Gemini 4 Argon form the Pareto frontier: DeepSeek at $0.04 per rollout, Opus adding ten points at $1.83, and Gemini edging ahead at $3.92.
For background on those models, see our posts on Claude Opus 5.5, DeepSeek V4.1 Flash and Gemini 4 Pro.
How to run it yourself #
The tasks run through Harbor straight from the Hugging Face repo, with no manual clone. Collinear's command, in outline:
harbor run --repo "https://huggingface.co/datasets/collinear-ai/cwe-bench" \
-d cwe-bench@2.0 -k 4 --agent <agent> --model <model> \
--env docker --allow-agent-host <model API host>
-k 4 runs four attempts per task, and the reported score is the mean reward over all 400 trials. The dataset is gated: accept its conditions on the dataset page first. Harbor fetches tasks with git, so save your token once with hf auth login --token $HF_TOKEN --add-to-git-credential.
Practical notes:
- Budget for Docker builds. Each task builds a vulnerable codebase.
- Costs per rollout differ by 100 times between the cheapest and priciest models in Collinear's table, so run a small sample first.
- Add your result by opening a PR on the model repo (open-weight) or on the leaderboard Space.
What is a CWE, and why 91 of them? #
CWE stands for Common Weakness Enumeration, the community catalogue of software weakness types, such as out-of-bounds writes, injection flaws and missing authorization checks. A single CWE can show up in many languages and frameworks. Covering 91 different CWEs in 100 tasks means almost every task tests a different class of bug, so a model cannot do well by learning one trick, such as adding input validation.
That breadth is also why the language mix matters. Go has 27 tasks and Rust only five. Memory-safety weaknesses appear mainly in the C and C++ tasks (24 together), while web-style flaws lean toward Java, JavaScript, TypeScript and Python. If your own stack is mostly one language, read the per-task results for that language, not only the overall mean.
What the scores mean for a team choosing an agent #
A few practical readings, which are our interpretation and not Collinear's:
- Treat pass@1 as the floor. If your pipeline can run the project's tests and an exploit check before accepting a patch, the pass@4 figure is closer to what you will see, because failed attempts are caught and retried.
- Price per rollout changes the answer. A model that is ten points better but costs about 46 times more per rollout (Opus at $1.83 versus DeepSeek at $0.04) may only make sense for the hard tail of issues. A tiered setup, with a cheap model first and an expensive one on failures, follows from the numbers.
- Never auto-merge. Even a passing patch only closed the exploits the verifier knew about. Human review stays necessary, and so does a regression suite.
- Keep the 19% in mind. The best model fails a fifth of the tasks on every attempt. In a real backlog those are likely the subtle bugs, which are also the dangerous ones.
Why a defensive benchmark matters #
Most cyber evaluations lean offensive: can a model write an exploit? See our coverage of ExploitBench and the VulnCheck data on AI-found bugs. CWE-bench flips the question. As Collinear puts it, "a defender has to anticipate the ways an attacker could get in, and get it right every single time." Fixing without regressions is a tough constraint, and it is the one real maintainers face.
The tasks are also tagged rl-environment. The same environments and verifiers can supply rewards for training defensive agents, not just for scoring them. That raises a point: if a benchmark doubles as training data, scores on the public set may drift upward without real gains. The held-out set is Collinear's guard against that.
If you deploy agents that edit code, pair evaluation with runtime controls. AgentBeam, from the explainx.ai team, stops AI agents before they take dangerous actions, which complements a benchmark that only measures how well they patch.
Limits and open questions #
- Vendor-run leaderboard. Collinear built the tasks and ran most evaluations.
- 100 tasks. A small set; one task is one percentage point.
- Mixed setups. The open-weight and all-model boards are separate. Agents, prompts and budgets may differ.
- Gated access. You must accept conditions to see the files.
- Memorization risk. The upstream-patch check helps but cannot prove a model has not seen a task.
- Names unverified. We did not independently confirm the model versions listed; they are as reported by Collinear.
Related reading #
Scores reflect Collinear AI's November 3, 2026 post and the dataset page at the time of reading.