---
type: "post"
id: "a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb"
slug: "a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb"
title: "A separate public benchmark analysis evaluated 23 models for correctness and code-review quality, with its top model passing every blocking check in 57.7% of attempts."
description: "The result makes the verification burden visible: the leading model cleared every blocking check in 57.7% of attempts."
retrieval_nugget: "The result makes the verification burden visible: the leading model cleared every blocking check in 57.7% of attempts."
published_at: "2026-10-14"
updated_at: "2026-10-14"
record_date: "2026-10-14"
date_kind: "published_at"
topics: []
source_urls: ["https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:13541afffe8332e1ca69a77ccab3de1f783d04c135ccfc2d53fd0d05c3dad5e7","reviewed_at":"2026-10-14T02:01:43.422Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The result makes the verification burden visible: the leading model cleared every blocking check in 57.7% of attempts.","observed_facts":[{"text":"Artificial Analysis tested 23 models on FrontierCode, and Claude Opus 5.5 ranked first with a 52.2% score.","source_urls":["https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"]},{"text":"The evaluation used the same Stirrup harness for each model, capped attempts at 250 turns, and ran three attempts per task on a 100-task Main set.","source_urls":["https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"]},{"text":"Nine tasks were unsolved by every model on every attempt.","source_urls":["https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"]}],"mechanism":"The mechanism is that FrontierCode treats mergeability as more than test success. Its checks include hidden tests, command checks, reverse and adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria.","why_now":"Why now: teams evaluating agents for pull requests need evidence about whether the resulting change clears review constraints, not only whether an agent can produce code.","implications":["The first implication is that teams should measure review-ready completion and retain human review rather than equating green tests with mergeable work."],"evidence_boundary":"These facts make the comparison interpretable within its setup, but the evidence boundary is this benchmark’s task set, harness, scoring rules, and tested configurations; it does not establish deployment outcomes across other repositories.","watch_conditions":["Watch whether later evaluations reproduce the gap between benchmark score and full blocking-check pass rate across different task sets and harnesses."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied for this benchmark result, so this begins a distinct analytical record."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb/","markdown":"https://newruntime.com/posts/a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb.md","json":"https://newruntime.com/posts/a-separate-public-benchmark-analysis-evaluated-23-models-6f6beb.json"}
---

# A separate public benchmark analysis evaluated 23 models for correctness and code-review quality, with its top model passing every blocking check in 57.7% of attempts.

## Retrieval answer

The result makes the verification burden visible: the leading model cleared every blocking check in 57.7% of attempts.

The result makes the verification burden visible: the leading model cleared every blocking check in 57.7% of attempts. Artificial Analysis tested 23 models on FrontierCode, and Claude Opus 5.5 ranked first with a 52.2% score. Those figures describe a particular benchmark run, not a universal rate of acceptable AI-generated code.

The mechanism is that FrontierCode treats mergeability as more than test success. Its checks include hidden tests, command checks, reverse and adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria. A change can therefore fail despite passing ordinary tests; the benchmark makes reviewability part of the measured task rather than assuming correctness alone is sufficient.

Why now: teams evaluating agents for pull requests need evidence about whether the resulting change clears review constraints, not only whether an agent can produce code. The evaluation used the same Stirrup harness for each model, capped attempts at 250 turns, and ran three attempts per task on a 100-task Main set. Nine tasks were unsolved by every model on every attempt. These facts make the comparison interpretable within its setup, but the evidence boundary is this benchmark’s task set, harness, scoring rules, and tested configurations; it does not establish deployment outcomes across other repositories. The first implication is that teams should measure review-ready completion and retain human review rather than equating green tests with mergeable work. Watch whether later evaluations reproduce the gap between benchmark score and full blocking-check pass rate across different task sets and harnesses.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)