{"type":"post","id":"artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4","slug":"artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4","title":"Artificial Analysis launched a leaderboard for Cognition's FrontierCode 1.1 benchmark.","description":"The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup.","retrieval_nugget":"The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup.","published_at":"2026-10-12","updated_at":"2026-10-12","record_date":"2026-10-12","date_kind":"published_at","topics":["claude"],"source_urls":["https://artificialanalysis.ai/evaluations/frontiercode"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:1cb553b161c2d319b5b041c65cd4e126cf526cd74ae8639a379724589d481967","reviewed_at":"2026-10-12T23:01:47.469Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The thesis is that the leaderboard makes coding-agent comparison more task- and cost-aware, while remaining bounded by its benchmark setup.","observed_facts":[{"text":"Artificial Analysis reports a 22-model comparison; Claude Opus 5.5 scores 52.2%, GPT-6.1 Sol 49.1%, and Claude Fable 5.1 49.0%.","source_urls":["https://artificialanalysis.ai/evaluations/frontiercode"]},{"text":"Artificial Analysis reports average evaluation cost per task and runs each task three times through its Stirrup harness, capped at 250 turns.","source_urls":["https://artificialanalysis.ai/evaluations/frontiercode"]}],"mechanism":"The mechanism is repeated evaluation against a maintainer-oriented acceptance bar: Artificial Analysis runs each task three times using its Stirrup harness, capped at 250 turns, and averages scores across the primary 100 tasks. The verifier combines tests and other checks, including scope checks and LLM review; failures on blocking checks score zero.","why_now":"Why now: this leaderboard puts quality and average task cost on the same comparison surface as agentic coding systems are being evaluated on work intended to be mergeable.","implications":["The first implication is that builders can use the published score-and-cost view to shortlist systems for their own testing, but should validate against their repositories and acceptance criteria."],"evidence_boundary":"The source describes the benchmark and implementation; it does not demonstrate that these checks predict deployment outcomes across teams or codebases.","watch_conditions":["Watch whether future benchmark releases broaden task coverage or publish evidence that the ranking is stable across harnesses and repeated runs."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied, so this benchmark leaderboard starts a distinct analytical record about task-based coding-agent comparison and cost."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4/","markdown":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4.md","json":"https://newruntime.com/posts/artificial-analysis-launched-a-leaderboard-for-cognition-s-b160a4.json"},"next_reads":[{"type":"topic","path":"/topics/claude/","url":"https://newruntime.com/topics/claude/","title":"Claude","reason":"Explore the claude topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/claude.md","json_url":"https://newruntime.com/topics/claude.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}