What happened
Artificial Analysis describes its Coding Agent Index as a composite measure of coding-agent performance on software engineering tasks, alongside cost, token usage and execution time. Version 2.1 combines three benchmark components: FrontierCode-AA 2.0, Terminal-Bench 4.1 and SWE-Atlas-QnA-AA. The announcement page identifies FrontierCode-AA 2.0 as software-engineering tasks, with 132 tasks, created by Cognition and implemented by Artificial Analysis. Terminal-Bench 4.1 covers agentic terminal use across 66 tasks and is by Laude Institute. SWE-Atlas-QnA-AA covers technical question answering across 124 tasks; it is by Scale AI, with grading adjusted by Artificial Analysis.
The methodology gives each of the three benchmarks equal weight in the Index. Within each benchmark, the score is the mean across three attempts per task. Artificial Analysis presents the combined score as useful for quick comparison, but cautions readers to consult the per-evaluation breakdown: two agents with similar Index values may perform differently on repository tasks, terminal workflows and rubric-based evaluations. The page includes a methodology link for scoring details and version history.
The index page organizes comparisons around more than benchmark score. It displays average API cost per task, total token use and active agent wall time per task, alongside views of performance against these measures. Its cost measure uses current per-token API pricing, including cache discounts and separate cache-write charges where applicable. The page explicitly frames this as pay-per-token API cost, not the price of consumer subscriptions or the full operational cost of deploying an agent. Engineering, infrastructure and supervision costs are outside that metric’s focus.
For runtime, Artificial Analysis measures how long the agent process is actively running on each task. The stated measure excludes environment startup, verifier or judge time, and other harness overhead. Token charts distinguish input, cached input and output; the page notes that prompt-cache hit rates can vary by provider routing and can materially affect effective cost. The visible comparison includes agent variants from multiple providers and harnesses, including Claude Code, Codex, Antigravity CLI, Muse Code, Kimi Code CLI and OpenCode. The page says there is no publicly available API for the index, time-per-task and cost-per-task displays.
What the archive adds
New Runtime covered Artificial Analysis’s Coding Agent Index v2.0 on 2026-10-23, when it combined three benchmarks and Claude Opus 5.5 in Claude Code led at 61.7. An earlier note on the FrontierCode 1.1 leaderboard dated 2026-10-14 also covered task- and cost-aware comparison. Version 2.1 is a new update to that known story, with revised benchmark components and measures; it is not a separate, independent benchmark launch.
Trend signal
This update reinforces the practical value of comparing agents by completed benchmark tasks alongside cost and runtime, rather than treating model identity or raw capability claims as sufficient. However, the index is a measurement framework, not evidence that cost per verified completed task captures every production constraint. This does not yet fit a pattern New Runtime tracks.
Connections to other events
New Runtime’s earlier coverage of Terminal-Bench 2.1 discussed a competing run against the benchmark. The present index incorporates Terminal-Bench 4.1 as one of three equally weighted components, alongside FrontierCode-AA 2.0 and SWE-Atlas-QnA-AA. These are related developments in coding-agent evaluation, but the index is not itself a report of a new agent capability.
Why it matters
Teams choosing a coding agent often compare headline scores while overlooking the workload, cost and time needed to produce them. This index makes those dimensions easier to inspect together. A setup that scores well but consumes substantially more API spend or active runtime may be the wrong fit for a high-volume workflow; conversely, a lower-cost option may be useful when its performance is strong on the tasks a team actually runs.
The useful decision is not to pick the overall leader by default. Use the per-benchmark breakdown to identify whether an agent is strong at repository work, terminal tasks or technical Q&A, then compare its cost and runtime on a representative workload. This turns the index into a shortlist and a prompt for local evaluation, rather than a substitute for one. Because the measurements use specific benchmark tasks and API pricing assumptions, a team should validate finalists against its own repositories, tools, review requirements and access model.
Limits and uncertainty
The Index is an equal-weight composite of three benchmarks, and its task mix does not establish how an agent will perform across every language, codebase or production workflow. Three attempts per task provide repeated observations, but the page does not establish that these captures all run-to-run variation. Cost figures are per-token API estimates under current pricing, not subscription rates or total operating costs; cache behavior can change effective spend. Runtime excludes startup, verification and other harness overhead. The displayed comparisons therefore support screening and investigation, not a guarantee of production cost, speed or quality.
Skill to build now
Build a small evaluation sheet for the coding tasks your team runs repeatedly. For each candidate agent, record task success, API cost, token use and active runtime, and keep repository work separate from terminal work and technical Q&A. Compare those local results with the Index’s component breakdowns before committing to a default setup.
Reader takeaway
Coding Agent Index v2.1 adds a useful, structured comparison across three kinds of coding-agent work, with cost, tokens and active runtime beside performance. Treat the composite score as a starting point: inspect its components, account for what the cost and time measures exclude, and verify likely candidates on your own tasks.