---
type: "post"
id: "swe-proof-turns-500-real-software-issues-into-3e10e9"
slug: "swe-proof-turns-500-real-software-issues-into-3e10e9"
title: "SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified"
description: "SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified proofs rather than judged only by test suites."
retrieval_nugget: "SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified proofs rather than judged only by test suites. Its evaluation reports that tests can miss counterexamples, while formal verification can improve resolution when the specification is correct. For teams assessing coding agents, the result sharpens the distinction between passing tests and establishing behavior."
published_at: "2026-11-20"
updated_at: "2026-11-20"
record_date: "2026-11-20"
date_kind: "published_at"
topics: ["claude"]
entities: ["SWE-Proof","Benchproofer","Claude Opus 4.8","SWE-bench Verified","SWE-bench Pro"]
related_patterns: ["scientific-agents-need-artifact-evidence"]
novelty_label: "new_fact"
what_changed: null
prior_refs: []
source_url: "https://arxiv.org/abs/2609.21190"
source_urls: ["https://arxiv.org/abs/2609.21190"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:swe-proof-turns-500-real-software-issues-into-3e10e9"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/swe-proof-turns-500-real-software-issues-into-3e10e9/","markdown":"https://newruntime.com/posts/swe-proof-turns-500-real-software-issues-into-3e10e9.md","json":"https://newruntime.com/posts/swe-proof-turns-500-real-software-issues-into-3e10e9.json"}
markdown_url: "https://newruntime.com/posts/swe-proof-turns-500-real-software-issues-into-3e10e9.md"
---

# SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified

## Retrieval answer

SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified proofs rather than judged only by test suites. Its evaluation reports that tests can miss counterexamples, while formal verification can improve resolution when the specification is correct. For teams assessing coding agents, the result sharpens the distinction between passing tests and establishing behavior.

## Summary
SWE-Proof turns 500 real software issues into a benchmark where patches are checked with machine-verified proofs rather than judged only by test suites. Its evaluation reports that tests can miss counterexamples, while formal verification can improve resolution when the specification is correct. For teams assessing coding agents, the result sharpens the distinction between passing tests and establishing behavior.

## What happened
In a paper submitted to arXiv on September 18, 2026, and revised on September 22, the authors introduce Benchproofer, a pipeline for turning a coding task with a known correct patch into one that can be formally verified. They apply it to SWE-bench Verified to create SWE-Proof, a set of 500 real issues. The paper says the approach also extends to SWE-bench Pro. The benchmark addresses a gap the authors identify in existing formal-verification work: prior tasks typically provide specifications as input, whereas real repository issues can span large codebases and describe intent in vague natural language.

Benchproofer creates a specification for the new code and summarizes existing functions that the code calls, representing those summaries with axioms. It admits an instance only when mechanical and adversarial gates agree. The abstract does not detail the individual gates, the formal language used, or the cost of running the pipeline, so the contribution described here is the benchmark and its construction method rather than a drop-in verification system for a production repository.

The authors evaluate Claude Opus 4.8. They report that a quarter of patches that pass tests admit counterexamples. A structured natural-language specification does not fix this problem, while a correct formal specification raises resolution from 85% to 95%. These results distinguish three separate things: a patch that passes a test suite, a specification that describes desired behavior, and a formal specification that is both faithful and machine-checkable.

The paper identifies specification writing as the difficult step. An agent tasked with writing its own specification gains nothing over an unaided baseline, and only 56% of those specifications pass the authors’ audit. The common failure is faithfulness: the specification constrains some required behavior but leaves other behavior unconstrained. Specification quality also tracks outcomes: specifications fail on 92% of unresolved instances, compared with 51% of resolved instances. The authors therefore frame faithful specification synthesis as an open problem, not a capability already solved by the benchmark.

## What the archive adds
New Runtime’s earlier coverage examined Claude-generated machine-checked proofs for 239 fuzz entry points across 123 open-source libraries ([October 23, 2026 note](https://newruntime.com/posts/anthropic-and-theorem-claude-wrote-machine-checked-no-9a7d2c/)). Another note covered EasyProver’s Lean results and its Rust-to-Lean benchmark ([October 31, 2026 note](https://newruntime.com/posts/easyprover-paper-off-the-shelf-frontier-model-proves-6f4bd9/)). SWE-Proof is a new benchmark result in the same broader area, focused on verifying patches for real software issues rather than standalone theorem proving or fuzz-entry-point proofs.

## Trend signal
This result supports the growing importance of evidence that can be inspected and mechanically checked when evaluating agents on technical work. It connects most closely to New Runtime’s [scientific-agent evidence pattern](https://newruntime.com/patterns/scientific-agents-need-artifact-evidence/): a score alone is less informative than an artifact others can examine and rerun. SWE-Proof provides a benchmark built around verified patches, although the paper’s reported outcomes do not establish that the pipeline is ready for routine engineering use.

## Why now
Coding-agent benchmark results increasingly depend on how correctness is measured. The authors’ comparison makes that measurement choice concrete: test suites accepted some patches for which they found counterexamples, while formal verification distinguished those from patches with correct formal specifications. That difference matters when teams interpret benchmark scores as evidence about agent reliability.

## Why it matters
For engineering teams, SWE-Proof is a reminder that test success and behavioral correctness are not interchangeable claims. Tests sample behavior; a counterexample can expose a failure they do not cover. Formal verification offers a stronger check for the behavior captured in a specification, but it does not remove the need to make that specification faithful. A formally checked patch can only establish what its formal contract actually says.

The reported specification results are as important as the headline improvement. When an agent writes its own contract, it may omit part of the requested behavior, and the paper’s audit found that this happened often. Teams considering proof-based checks should therefore treat specification authorship and review as part of the engineering work, not assume the agent can safely define its own success conditions. The benchmark gives researchers a way to study that bottleneck on real issues; it does not show that every issue can already be specified or verified economically.

## Limits and uncertainty
These are results reported by the paper for Claude Opus 4.8 and the authors’ benchmark construction. The abstract does not establish how results generalize to other models, repositories, languages, or issue types, nor does it report the runtime or human effort required to create and audit specifications. The 500 benchmark instances are drawn from SWE-bench Verified; the reported figures should not be read as a measured rate of defects in production code. The authors also describe faithful specification synthesis as unresolved. The benchmark’s broader reproducibility and independent replication are not established by the reported abstract.

## Skill to build now
When reviewing an agent-generated patch, separate the claim “the tests pass” from the claim “the requested behavior is fully specified.” For a high-risk change, write down the behavior the patch must preserve and inspect whether the tests and assertions cover each part, including relevant edge cases. This is not formal verification, but it makes the contract gap visible before a team invests in proof tooling.

## Reader takeaway
SWE-Proof suggests that coding-agent evaluation needs stronger evidence than passing tests alone: a quarter of test-passing patches in the reported evaluation had counterexamples. Formal checks helped when the specification was correct, but writing a faithful specification remains the hard part. The benchmark makes that gap measurable; it does not make it disappear.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)
