---
type: "post"
id: "artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4"
slug: "artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4"
title: "Artificial Analysis launched a leaderboard for Cognition's FrontierCode 1.1 benchmark."
description: "The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup."
retrieval_nugget: "The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup."
published_at: "2026-10-12"
updated_at: "2026-10-12"
record_date: "2026-10-12"
date_kind: "published_at"
topics: ["claude"]
source_urls: ["https://artificialanalysis.ai/evaluations/frontiercode"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:1cb553b161c2d319b5b041c65cd4e126cf526cd74ae8639a379724589d481967","reviewed_at":"2026-10-12T23:01:47.469Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup.","observed_facts":[{"text":"Artificial Analysis reports a 22-model comparison; Claude Opus 5.5 scores 52.2%, GPT-6.1 Sol 49.1%, and Claude Fable 5.1 49.0%.","source_urls":["https://artificialanalysis.ai/evaluations/frontiercode"]},{"text":"Artificial Analysis reports average evaluation cost per task and runs each task three times through its Stirrup harness, capped at 250 turns.","source_urls":["https://artificialanalysis.ai/evaluations/frontiercode"]}],"mechanism":"The mechanism is repeated evaluation against a maintainer-oriented acceptance bar: Artificial Analysis runs each task three times using its Stirrup harness, capped at 250 turns, and averages scores across the primary 100 tasks. The verifier combines tests and other checks, including scope checks and LLM review; failures on blocking checks score zero.","why_now":"Why now: this leaderboard puts quality and average task cost on the same comparison surface as agentic coding systems are being evaluated on work intended to be mergeable.","implications":["The first implication is that builders can use the published score-and-cost view to shortlist systems for their own testing, but should validate against their repositories and acceptance criteria."],"evidence_boundary":"The source describes the benchmark and implementation; it does not demonstrate that these checks predict deployment outcomes across teams or codebases.","watch_conditions":["Watch whether future benchmark releases broaden task coverage or publish evidence that the ranking is stable across harnesses and repeated runs."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied, so this benchmark leaderboard starts a distinct analytical record about task-based coding-agent comparison and cost."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4/","markdown":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4.md","json":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4.json"}
---

# Artificial Analysis launched a leaderboard for Cognition's FrontierCode 1.1 benchmark.

## Retrieval answer

The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup.

The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup. Artificial Analysis reports a 22-model comparison, with Claude Opus 5.5 at 52.2%, GPT-6.1 Sol at 49.1%, and Claude Fable 5.1 at 49.0%. It also displays average evaluation cost per task. These observed facts make the page useful as a comparative instrument, not as a general verdict on which model will perform best in every repository.

The mechanism is repeated evaluation against a maintainer-oriented acceptance bar: Artificial Analysis runs each task three times using its Stirrup harness, capped at 250 turns, and averages scores across the primary 100 tasks. The verifier combines tests and other checks, including scope checks and LLM review; failures on blocking checks score zero. This turns a coding result into a measure of whether repository changes clear a defined set of acceptance conditions, rather than merely whether code was produced. The source describes the benchmark and implementation; it does not demonstrate that these checks predict deployment outcomes across teams or codebases.

Why now: this leaderboard puts quality and average task cost on the same comparison surface as agentic coding systems are being evaluated on work intended to be mergeable. The first implication is that builders can use the published score-and-cost view to shortlist systems for their own testing, but should validate against their repositories and acceptance criteria. Watch whether future benchmark releases broaden task coverage or publish evidence that the ranking is stable across harnesses and repeated runs.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)