What happened
Collinear AI describes CWE-bench as a benchmark for testing the limits of frontier cyber capabilities. In v2, the public set contains 100 tasks spanning 91 different CWE categories. Each task gives an agent a checkout of a real open-source repository and a short instruction: find and fix a security weakness without breaking existing functionality. The setup is the same as before, according to the authors.
Scoring is fully programmatic and deterministic, so the authors say results can be reproduced without extra cost. They say LLM judge panels helped curate the programmatic checks for hardness and fairness, and that every task was also reviewed by hand. Their quality review asked whether instructions sound realistic, whether graded behavior can be inferred from those instructions, whether the verifier checks behavior rather than a particular fix structure, and whether the starter code fails while the reference fix passes. They also checked that existing functionality still works after the fix and that verifiers reject memorized solutions such as known upstream patches.
The authors maintain a held-out set to track progress on tasks models have not seen. The public 100-task set is open, but access to the held-out evaluation requires contacting cyber@collinear.ai. The public tasks, grader and harness are open; the dataset itself is gated, and users must accept its access conditions on the Hugging Face dataset page before running it.
Results are tracked in two places: a dataset-page leaderboard for open-weight models, where labs and community members can submit results through a pull request, and a leaderboard Space for the full comparison, including API-only models such as Gemini, Claude and GPT. Of 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1 and 81.0% pass@4, just ahead of Claude Opus 5.5 at 67.3% and 78.0%. At least one model solves 95 of the 100 tasks. Still, the leader fails 19% of the set on every attempt, according to the post.
The reported cost frontier includes DeepSeek V4.1 Flash (1017), Claude Opus 5.5 and Gemini 4 Argon. DeepSeek reaches 57.0% pass@1 at $0.04 per rollout; Opus adds ten percentage points at $1.83, and Gemini edges ahead at $3.92. Each task is a self-contained Harbor task: a Dockerized environment, an instruction and a deterministic verifier. Teams can run it from the Hugging Face repository with Harbor; the documented command uses four attempts per task, for a mean reward across 400 trials. The tasks are also tagged as an RL environment, so the same environments and verifiers can provide rewards for training defensive agents.
What the archive adds
This is a new benchmark release, not a restatement of earlier coverage. We have covered adjacent evaluation work in a public analysis of 23 models and code-review quality and ReviewBench. Those concern code correctness and review, not this benchmark's security-fix tasks; CWE-bench adds a separate test surface and published cost figures.
Trend signal
No supplied archive pattern is directly linked to this release. The benchmark nevertheless makes a practical distinction visible: a plausible patch is not sufficient when an evaluator must also establish that the vulnerability is fixed and existing behavior remains intact.
Why it matters
For teams building defensive agents, the benchmark offers a shared workload on which to compare models and harnesses rather than relying on demonstrations or self-reported success. Deterministic verifiers reduce ambiguity between runs, and the task design explicitly checks that fixes preserve existing behavior. That makes the release useful for finding failure modes before deploying an agent against repositories that matter.
The results also argue against choosing on a single leaderboard rank. DeepSeek V4.1 Flash is much cheaper per rollout in the reported comparison, while the top result belongs to Gemini 4 Argon; the difference between pass@1 and pass@4 shows that repeated attempts affect measured performance. Teams should compare success rates and total evaluation cost under their own attempt budget, and inspect whether the benchmark's task mix resembles their security work. The public set can help with that evaluation, but it is not evidence that a model is secure or reliable in every repository.
Limits and uncertainty
The post reports results for 11 models but does not specify in the supplied material the full per-model setup, model versions beyond the names shown, or a confidence interval for the scores. It does not establish how these results transfer to private codebases, other vulnerability distributions or deployment conditions. The held-out tasks are not publicly available through the open set; access requires contacting the authors. Costs are reported per rollout, and the post does not provide a team's total operating cost or establish that the listed cost-quality frontier will hold for other prompts, harnesses or attempt budgets.
Skill to build now
Run a small, controlled evaluation before choosing a defensive coding agent. After accepting the dataset's access conditions and configuring Harbor and the required model API access, run CWE-bench v2 with a fixed model, harness and attempt budget. Record pass@1, pass@4 where applicable, and cost per rollout; inspect failed tasks as well as the aggregate score. Keeping those settings constant makes a comparison with another model more informative than comparing leaderboard numbers produced under unknown conditions.
Reader takeaway
CWE-bench v2 turns security-agent evaluation into a runnable, deterministic task set, with public results that expose both repeated-attempt performance and cost trade-offs. Use it as one reproducible check on defensive agents—not as a substitute for testing on your own code or for broader security review.