Vercel’s DeepsecBench is useful because it refuses to make security-agent evaluation a simple leaderboard story.
The benchmark measures models on vulnerability discovery in application code, but the report includes recall, precision, cost, and total time. The score is a recall-weighted F2 metric. That choice is explicit: missed vulnerabilities remain unfixed, while false positives do not make the codebase less secure.
The construction is also important. Vercel says the benchmark runs on an open-source codebase at a commit before a large set of vulnerabilities was fixed. It selected 50 entry-point files and built a golden set of 231 human-judged findings. Each model is run three times, and the published result is the median. The repository, commit, files, and findings are kept secret to reduce training leakage.
What the numbers say
The sample leaderboard shows why teams need a policy, not a winner:
- GPT-5.6 Sol at xhigh scores 35.58, costs $55.98, and takes 3:39;
- Claude Opus 5 at medium scores 28.36, costs $31.96, and takes 47 minutes;
- Kimi K3 at high scores 17.56, costs $12.38, and takes 1:59;
- Grok 4.5 at high scores 15.58, costs $5.60, and takes 1:24.
Higher spend does not buy proportional improvement. Lower-cost and open-weight options do not replace the best frontier run, but they can change how often a team scans and which repos receive deeper passes.
New Runtime Read
This is the right shape for agent economics: completed defensive work per dollar, per hour, and per missed issue.
Security teams will not choose a model once. They will choose a scanning portfolio: cheap frequent passes, deeper scheduled scans, targeted reruns after sensitive diffs, and escalation when a lower-cost model finds something that needs validation. DeepsecBench makes that portfolio conversation more concrete.