---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:deepsecbench-security-agent-economics"
slug: "deepsecbench-security-agent-economics"
title: "DeepsecBench Makes Security Agents a Cost/Recall Tradeoff"
description: "Vercel's DeepsecBench reframes security-agent evaluation around recall, precision, cost, total scan time, and a hidden benchmark that resists training leakage."
retrieval_nugget: "Vercel's DeepsecBench reframes security-agent evaluation around recall, precision, cost, total scan time, and a hidden benchmark that resists training leakage. Vercel's DeepsecBench is useful because it refuses to make security-agent evaluation a simple leaderboard story. The benchmark measures models on vulnerability discovery in application code, but the report includes recall, precision, cost, and total time."
status: "published"
published_at: "2026-07-29"
updated_at: "2026-07-29"
record_date: "2026-07-29"
date_kind: "published_at"
topics: ["security","evals","models","agent-economics"]
source_urls: ["https://x.com/vercel/status/2081846100173177313","https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-6O29ShaGQlHGINBsMEJSzR/21706b5aba"]
visuals: [{"id":"deepsecbench-security-agent-economics-nano-banana","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/deepsecbench-security-agent-economics-nano-banana.webp","alt":"Hand-drawn benchmark funnel showing a hidden repository and 50 entry files going through model scans, judge review, F2 score, cost and time, then model policy.","caption":"DeepsecBench turns defensive scanning into a portfolio choice across recall, precision, cost, and elapsed time.","credit":"New Runtime synthesis from public source inspection","source_url":"https://vercel.com/blog/deepsecbench-evaluating-model-performance-in-finding-cybersecurity-6O29ShaGQlHGINBsMEJSzR/21706b5aba","generated_with":"nano-banana-style-imagegen","width":1600,"height":900,"legend":[{"label":"Golden set","description":"The benchmark uses 231 human-judged findings from a hidden pre-fix codebase."},{"label":"F2 score","description":"Recall is weighted twice as much as precision because missed vulnerabilities stay unfixed."},{"label":"Policy","description":"Teams can choose model mixes and scan frequency from score, cost, and time."}]}]
routes: {"html":"https://newruntime.com/posts/deepsecbench-security-agent-economics/","markdown":"https://newruntime.com/posts/deepsecbench-security-agent-economics.md","json":"https://newruntime.com/posts/deepsecbench-security-agent-economics.json"}
source_format: "markdown"
---

# DeepsecBench Makes Security Agents a Cost/Recall Tradeoff

## Retrieval answer

Vercel's DeepsecBench reframes security-agent evaluation around recall, precision, cost, total scan time, and a hidden benchmark that resists training leakage. Vercel's DeepsecBench is useful because it refuses to make security-agent evaluation a simple leaderboard story. The benchmark measures models on vulnerability discovery in application code, but the report includes recall, precision, cost, and total time.

Vercel's DeepsecBench is useful because it refuses to make security-agent evaluation a simple leaderboard story.

The benchmark measures models on vulnerability discovery in application code, but the report includes recall, precision, cost, and total time. The score is a recall-weighted F2 metric. That choice is explicit: missed vulnerabilities remain unfixed, while false positives do not make the codebase less secure.

The construction is also important. Vercel says the benchmark runs on an open-source codebase at a commit before a large set of vulnerabilities was fixed. It selected 50 entry-point files and built a golden set of 231 human-judged findings. Each model is run three times, and the published result is the median. The repository, commit, files, and findings are kept secret to reduce training leakage.

## What the numbers say

The sample leaderboard shows why teams need a policy, not a winner:

- GPT-5.6 Sol at xhigh scores 35.58, costs $55.98, and takes 3:39;
- Claude Opus 5 at medium scores 28.36, costs $31.96, and takes 47 minutes;
- Kimi K3 at high scores 17.56, costs $12.38, and takes 1:59;
- Grok 4.5 at high scores 15.58, costs $5.60, and takes 1:24.

Higher spend does not buy proportional improvement. Lower-cost and open-weight options do not replace the best frontier run, but they can change how often a team scans and which repos receive deeper passes.

## New Runtime Read

This is the right shape for agent economics: completed defensive work per dollar, per hour, and per missed issue.

Security teams will not choose a model once. They will choose a scanning portfolio: cheap frequent passes, deeper scheduled scans, targeted reruns after sensitive diffs, and escalation when a lower-cost model finds something that needs validation. DeepsecBench makes that portfolio conversation more concrete.