What happened
OpenHack says PentestBench evaluates how well models and harnesses find vulnerabilities in complete web applications built with modern frameworks. Version 1, published November 16, 2026, covers 17 applications and 761 planted vulnerabilities. The application set spans Next.js, Express, Fastify, SvelteKit, Django, Flask, FastAPI, Ruby on Rails, Laravel, Spring Boot, and Gin. It includes vulnerability-chain scenarios, such as a weak session-signing key enabling an authorization bypass, or cross-site scripting combined with readable session cookies enabling session theft.
The benchmark is white-box: every evaluated tool receives the application source code, while the reference vulnerability catalog stays sealed. OpenHack reports three measures. Recall is the share of planted vulnerabilities found; precision is the share of reported findings that match a planted vulnerability; catalog F1 is their harmonic mean and the ranking metric. The page says the leaderboard ranks each scanner configuration separately and presents results across all 17 applications.
OpenHack v2 Harness with Mythos 5.2 leads the listed table with 277 matched vulnerabilities out of 521 reports: 36.40% recall, 53.17% precision, and 43.21% catalog F1. Claude Code /security-review with Mythos 5.2 + Haiku records 200 matches from 319 reports, 26.28% recall, 62.70% precision, and 37.04% F1. Claude Code /security-review with Opus 5.5 reports 182 matches from 264, for 23.92% recall, 68.94% precision, and 35.51% F1. Claude Security configurations have higher recall but lower precision: Opus 5.5 + Sonnet reaches 40.74% recall and 30.54% precision; Mythos 5.2 + Sonnet reaches 42.97% recall and 26.24% precision. Semgrep CE 1.185.0 with security-audit rules records 19 matches from 50 reports, 2.50% recall, 38.00% precision, and 4.69% F1.
OpenHack describes this as the first version of its corpus and says it created the benchmark to evaluate its own always-on AI security engineer. It plans to open-source the 17-application corpus. The company says future versions will include applications deployed in cloud environments and evaluate vulnerability reachability and false-positive identification. Black-box pentesting results are also described as forthcoming. OpenHack says it is still awaiting access to Daybreak Red from OpenAI to include GPT-6 Astra; no OpenAI model appears in this leaderboard.
What the archive adds
New Runtime’s earlier coverage has examined CWE-bench v2 (November 3, 2026), which evaluates agents fixing repository vulnerabilities, and SecurityScorecard’s open TPRM benchmark (October 31, 2026), focused on third-party risk and attack resistance. PentestBench is a new benchmark on a known theme: evaluating security systems, here against planted flaws in complete web applications and model-plus-harness configurations.
Trend signal
This does not yet fit a pattern New Runtime tracks. PentestBench adds another concrete way to compare security-agent configurations, but its first release is a publisher-built benchmark and does not establish a broader shift in how teams verify agent output.
Why now
OpenHack says it built PentestBench because existing evaluations did not meet its goal of measuring vulnerability discovery in complete, modern web applications, including chained weaknesses, rather than isolated snippets or functions. That framing explains the benchmark’s emphasis on application-level scenarios. Its planned cloud-deployed apps and reachability checks point toward the company’s stated next steps, not capabilities already measured in v1.
Why it matters
Security teams need to know whether an agent finds meaningful vulnerabilities without flooding reviewers with weak reports. PentestBench makes that trade-off visible: the OpenHack configuration has the highest F1 on the published table, while some Claude Security configurations find a larger share of planted flaws but produce many more reports. A single leaderboard rank would conceal that operational difference.
The benchmark is also a reminder that the evaluated unit is a configuration, not a model in isolation. Harness, model, and review setup vary together across entries, so teams should not infer that one model alone caused a score. The results can inform what to test locally, especially for teams using similar white-box workflows. They cannot establish how these systems perform against unplanted flaws, in black-box settings, or across a broader population of production applications. Those are consequential gaps for anyone deciding whether an agent can be trusted in a security process.
Limits and uncertainty
These are early results from 17 applications and 761 planted vulnerabilities, not a demonstrated measure of performance across production systems. OpenHack’s own harness leads, and the publisher says the benchmark was created to evaluate its own security engineer; the page does not report an independent replication. Because the catalog remains sealed, the results depend on OpenHack’s labeling and scoring process. The corpus is planned for open-source release, rather than described as already available. Version 1 is white-box only, and the page says black-box results are forthcoming. It also does not provide results for OpenAI models, which OpenHack says it has not yet been able to evaluate.
Reader takeaway
PentestBench v1 offers a useful, measurable comparison of security-agent configurations, but its scores are not a readiness certificate. Read recall and precision alongside F1, and treat the publisher’s own leading result as an invitation to reproduce the evaluation—not as proof that the same ranking will hold in your applications.