Artificial Analysis reports a 22-model comparison, with Claude Opus 5.5 at 52.2%, GPT-6.1 Sol at 49.1%, and Claude Fable 5.1 at 49.0%. It also displays average evaluation cost per task. These observed facts make the page useful as a comparative instrument, not as a general verdict on which model will perform best in every repository.
The mechanism is repeated evaluation against a maintainer-oriented acceptance bar: Artificial Analysis runs each task three times using its Stirrup harness, capped at 250 turns, and averages scores across the primary 100 tasks. The verifier combines tests and other checks, including scope checks and LLM review; failures on blocking checks score zero. This turns a coding result into a measure of whether repository changes clear a defined set of acceptance conditions, rather than merely whether code was produced. The source describes the benchmark and implementation; it does not demonstrate that these checks predict deployment outcomes across teams or codebases.
Why now: this leaderboard puts quality and average task cost on the same comparison surface as agentic coding systems are being evaluated on work intended to be mergeable. The first implication is that builders can use the published score-and-cost view to shortlist systems for their own testing, but should validate against their repositories and acceptance criteria. Watch whether future benchmark releases broaden task coverage or publish evidence that the ranking is stable across harnesses and repeated runs.