Claude Opus 5.5 ranked first with a 52.2% score and passed every blocking check in 57.7% of attempts. The evaluation used the same open-source Stirrup harness for each model, a 250-turn limit, and three attempts per task across the 100-task Main set. Nine tasks were not solved by any model on any attempt. FrontierCode includes hidden tests, command checks, reverse and adaptive tests, limits on files and diff size, and an LLM review against maintainer criteria. The results are specific to this benchmark and setup; they do not establish expected performance across all repositories or engineering teams.
Raw SignalOct 14, 20261 min read1 source
A separate public benchmark analysis evaluated 23 models for correctness and code-review quality, with its top model passing every blocking check in 57.7% of attempts.
In one minute
Artificial Analysis published results for 23 models tested on FrontierCode, a benchmark that evaluates repository changes for correctness and code-review quality.