---
type: "raw_signal"
id: "a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d"
slug: "a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d"
title: "A separate public benchmark analysis evaluated 23 models for correctness and code-review quality, with its top model passing every blocking check in 57.7% of attempts."
description: "Artificial Analysis published results for 23 models tested on FrontierCode, a benchmark that evaluates repository changes for correctness and code-review quality."
retrieval_nugget: "Artificial Analysis published results for 23 models tested on FrontierCode, a benchmark that evaluates repository changes for correctness and code-review quality."
observed_at: "2026-10-14"
record_date: "2026-10-14"
date_kind: "observed_at"
why_it_matters: "Source: https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"
novelty: "notable"
verification_level: "source-linked"
signal_type: "research"
source_platform: "bestcadpapers.com"
topics: []
entities: ["Artificial Analysis","FrontierCode","Claude Opus 5.5","Stirrup"]
related_patterns: []
source_url: "https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"
source_urls: ["https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "signal:a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/signals/a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d/","markdown":"https://newruntime.com/signals/a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d.md","json":"https://newruntime.com/signals/a-separate-public-benchmark-analysis-evaluated-23-models-a42a7d.json"}
---

# A separate public benchmark analysis evaluated 23 models for correctness and code-review quality, with its top model passing every blocking check in 57.7% of attempts.

## Retrieval answer

Artificial Analysis published results for 23 models tested on FrontierCode, a benchmark that evaluates repository changes for correctness and code-review quality.

Artificial Analysis published results for 23 models tested on FrontierCode, a benchmark that evaluates repository changes for correctness and code-review quality. Claude Opus 5.5 ranked first with a 52.2% score and passed every blocking check in 57.7% of attempts. The evaluation used the same open-source Stirrup harness for each model, a 250-turn limit, and three attempts per task across the 100-task Main set. Nine tasks were not solved by any model on any attempt. FrontierCode includes hidden tests, command checks, reverse and adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria. The results are specific to this benchmark and setup; they do not establish expected performance across all repositories or engineering teams.

Source: https://bestcadpapers.com/software-qa-development/frontiercode-a-maintainer-s-checklist-for-reviewing-ai-pull-requests

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Raw Signals](https://newruntime.com/signals.json): Open every source-linked raw signal.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)