Field noteNov 17, 20265 min read1 source

OpenHack publishes PentestBench v1, an open-source web pentest benchmark

In one minute

OpenHack published PentestBench v1, a white-box benchmark of model-and-harness configurations searching for web vulnerabilities in 17 applications containing 761 planted flaws. Its results put OpenHack’s own configuration first, while showing how recall and precision can diverge. For teams evaluating security agents, the release offers a concrete comparison framework—but not an independent verdict on real-world performance.

What changed: OpenHack has published PentestBench v1, a white-box web penetration-testing benchmark spanning 17 apps and 761 planted vulnerabilities, with early model and scanner results.

What happened

OpenHack says PentestBench evaluates how well models and harnesses find vulnerabilities in complete web applications built with modern frameworks. Version 1, published November 16, 2026, covers 17 applications and 761 planted vulnerabilities. The application set spans Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. It includes vulnerability-chain scenarios, such as a weak session-signing key enabling an authorization bypass, or cross-site scripting combined with readable session cookies enabling session theft.

The benchmark is white-box: every evaluated tool receives the application source code, while the reference vulnerability catalog stays sealed. OpenHack reports three measures. Recall is the share of planted vulnerabilities found; precision is the share of reported findings that match a planted vulnerability; catalog F1 is their harmonic mean and the ranking metric. The page says the leaderboard ranks each scanner configuration separately and presents results across all 17 applications.

OpenHack v2 Harness with Mythos 5.2 leads the listed table with 277 matched vulnerabilities out of 521 reports: 36.40% recall, 53.17% precision, and 43.21% catalog F1. Claude Code /security-review with Mythos 5.2 + Haiku records 200 matches from 319 reports, 26.28% recall, 62.70% precision, and 37.04% F1. Claude Code /security-review with Opus 5.5 reports 182 matches from 264, for 23.92% recall, 68.94% precision, and 35.51% F1. Claude Security configurations have higher recall but lower precision: Opus 5.5 + Sonnet reaches 40.74% recall and 30.54% precision; Mythos 5.2 + Sonnet reaches 42.97% recall and 26.24% precision. Semgrep CE 1.185.0 with security-audit rules records 19 matches from 50 reports, 2.50% recall, 38.00% precision, and 4.69% F1.

OpenHack describes this as the first version of its corpus and says it created the benchmark to evaluate its own always-on AI security engineer. It plans to open-source the 17-application corpus. The company says future versions will include applications deployed in cloud environments and evaluate vulnerability reachability and false-positive identification. Black-box pentesting results are also described as forthcoming. OpenHack says it is still awaiting access to Daybreak Red from OpenAI to include GPT-6 Astra; no OpenAI model appears in this leaderboard.

What the archive adds

New Runtime’s earlier coverage has examined CWE-bench v2 (November 3, 2026), which evaluates agents fixing repository vulnerabilities, and SecurityScorecard’s open TPRM benchmark (October 31, 2026), focused on third-party risk and attack resistance. PentestBench is a new benchmark on a known theme: evaluating security systems, here against planted flaws in complete web applications and model-plus-harness configurations.

Trend signal

This does not yet fit a pattern New Runtime tracks. PentestBench adds another concrete way to compare security-agent configurations, but its first release is a publisher-built benchmark and does not establish a broader shift in how teams verify agent output.

Why now

OpenHack says it built PentestBench because existing evaluations did not meet its goal of measuring vulnerability discovery in complete, modern web applications, including chained weaknesses, rather than isolated snippets or functions. That framing explains the benchmark’s emphasis on application-level scenarios. Its planned cloud-deployed apps and reachability checks point toward the company’s stated next steps, not capabilities already measured in v1.

Why it matters

Security teams need to know whether an agent finds meaningful vulnerabilities without flooding reviewers with weak reports. PentestBench makes that trade-off visible: the OpenHack configuration has the highest F1 on the published table, while some Claude Security configurations find a larger share of planted flaws but produce many more reports. A single leaderboard rank would conceal that operational difference.

The benchmark is also a reminder that the evaluated unit is a configuration, not a model in isolation. Harness, model, and review setup vary together across entries, so teams should not infer that one model alone caused a score. The results can inform what to test locally, especially for teams using similar white-box workflows. They cannot establish how these systems perform against unplanted flaws, in black-box settings, or across a broader population of production applications. Those are consequential gaps for anyone deciding whether an agent can be trusted in a security process.

Limits and uncertainty

These are early results from 17 applications and 761 planted vulnerabilities, not a demonstrated measure of performance across production systems. OpenHack’s own harness leads, and the publisher says the benchmark was created to evaluate its own security engineer; the page does not report an independent replication. Because the catalog remains sealed, the results depend on OpenHack’s labeling and scoring process. The corpus is planned for open-source release, rather than described as already available. Version 1 is white-box only, and the page says black-box results are forthcoming. It also does not provide results for OpenAI models, which OpenHack says it has not yet been able to evaluate.

Reader takeaway

PentestBench v1 offers a useful, measurable comparison of security-agent configurations, but its scores are not a readiness certificate. Read recall and precision alongside F1, and treat the publisher’s own leading result as an invitation to reproduce the evaluation—not as proof that the same ranking will hold in your applications.

Evidence / sources

  1. 1pentestbench.com
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicClaudeExplore the claude topic hub.
  2. 02topicClaude CodeExplore the claude-code topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04sourcesSourcesSee the original sources this archive cites.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract