Its implementation runs the primary 100-task set from a 150-task benchmark, with each task attempted three times through Artificial Analysis’ Stirrup harness, capped at 250 turns. The verifier checks classical tests, command checks, reverse-classical tests, adaptive tests, scope, and an LLM prompt review; a failure on any blocking check, or a trajectory flagged for solution lookup, scores zero.
The leaderboard covers 22 models. Claude Opus 5.5 (max with fallback) leads at 52.2%, followed by GPT-6.1 Sol (max) at 49.1% and Claude Fable 5.1 (max with fallback) at 49.0%. Artificial Analysis also reports average evaluation cost per task. The results are specific to this benchmark implementation, harness, and scoring setup; they do not establish a universal ranking of coding agents.