---
type: "post"
id: "artificial-analysis-publishes-per-benchmark-results-with-the-03859f"
slug: "artificial-analysis-publishes-per-benchmark-results-with-the-03859f"
title: "Artificial Analysis publishes per-benchmark results with the index and documents task construction, scoring, and model handling in its methodology ."
description: "Thesis: A benchmark aggregate becomes more useful when its component tasks and scoring rules are inspectable, but transparency alone does not establish that its mix matches a team's deployment needs."
retrieval_nugget: "Thesis: A benchmark aggregate becomes more useful when its component tasks and scoring rules are inspectable, but transparency alone does not establish that its mix matches a team's deployment needs."
published_at: "2026-10-14"
updated_at: "2026-10-14"
record_date: "2026-10-14"
date_kind: "published_at"
topics: []
source_urls: ["https://artificialanalysis.ai/methodology/decision-index"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:artificial-analysis-publishes-per-benchmark-results-with-the-03859f"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:6eaff04470e7b1590e6ba533ed90a38365c5e575c54fab1e8e5df3d50bb41218","reviewed_at":"2026-10-14T02:01:43.422Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"A benchmark aggregate becomes more useful when its component tasks and scoring rules are inspectable, but transparency alone does not establish that its mix matches a team's deployment needs.","observed_facts":[{"text":"Artificial Analysis says the Decision Index is a weighted average of four evaluations, each assigned 25%: AA-Omniscience-Decision, HLE-MultiChoice, When2Call, and WildGuardTest.","source_urls":["https://artificialanalysis.ai/methodology/decision-index"]},{"text":"The methodology specifies that AA-Omniscience-Decision contains 500 private questions with four answer choices plus “I don't know,” and splits its contribution between accuracy and non-hallucination rate.","source_urls":["https://artificialanalysis.ai/methodology/decision-index"]},{"text":"It says HLE-MultiChoice uses 513 text-only questions from Humanity's Last Exam.","source_urls":["https://artificialanalysis.ai/methodology/decision-index"]}],"mechanism":"The score combines distinct capabilities—knowledge and hallucination, reasoning, tool calling, and classification—under fixed equal weights. The page also describes differences in response handling: decision models return choices and probabilities, while language models receive a fixed prompt and are constrained to structured outputs.","why_now":"Teams comparing model endpoints need to know which tasks and scoring conventions shape a benchmark before using it to shortlist candidates.","implications":["Teams can use the index as an initial shortlist, then test leading candidates against their own labels, traffic, and error costs."],"evidence_boundary":"The available source documents the benchmark's composition and stated scoring procedure; it does not establish how representative the tasks are of a particular production workload or whether the ranking predicts outcomes there.","watch_conditions":["Watch for teams publishing evaluations that compare the index's shortlist with results on their own labeled workloads and error-cost profiles."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied for this exact Decision Index methodology disclosure; it warrants a distinct record focused on benchmark composition and score interpretation."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/artificial-analysis-publishes-per-benchmark-results-with-the-03859f/","markdown":"https://newruntime.com/posts/artificial-analysis-publishes-per-benchmark-results-with-the-03859f.md","json":"https://newruntime.com/posts/artificial-analysis-publishes-per-benchmark-results-with-the-03859f.json"}
---

# Artificial Analysis publishes per-benchmark results with the index and documents task construction, scoring, and model handling in its methodology .

## Retrieval answer

Thesis: A benchmark aggregate becomes more useful when its component tasks and scoring rules are inspectable, but transparency alone does not establish that its mix matches a team's deployment needs.

Thesis: A benchmark aggregate becomes more useful when its component tasks and scoring rules are inspectable, but transparency alone does not establish that its mix matches a team's deployment needs.

Observed facts: Artificial Analysis says the Decision Index is a weighted average of four evaluations, each assigned 25%: AA-Omniscience-Decision, HLE-MultiChoice, When2Call, and WildGuardTest. The methodology specifies that AA-Omniscience-Decision contains 500 private questions with four answer choices plus “I don't know,” and splits its contribution between accuracy and non-hallucination rate. It says HLE-MultiChoice uses 513 text-only questions from Humanity's Last Exam.

Mechanism: The score combines distinct capabilities—knowledge and hallucination, reasoning, tool calling, and classification—under fixed equal weights. The page also describes differences in response handling: decision models return choices and probabilities, while language models receive a fixed prompt and are constrained to structured outputs. That disclosure lets a reader inspect how the aggregate is assembled rather than treating the headline score as self-explanatory.

Why now: Teams comparing model endpoints need to know which tasks and scoring conventions shape a benchmark before using it to shortlist candidates. The published methodology provides that inspection layer; it does not show that the index predicts performance on any particular team's workload.

First implication: Teams can use the index as an initial shortlist, then test leading candidates against their own labels, traffic, and error costs.

The practical distinction is between comparability and fit. Equal weighting makes the index's combination legible, but a team's priorities may weight tool-use failures, abstention, or classification errors differently. The public methodology describes construction and scoring; it does not, by itself, establish external validity for a specific application or provide evidence about how the aggregate correlates with production outcomes.

Evidence boundary: The available source documents the benchmark's composition and stated scoring procedure; it does not establish how representative the tasks are of a particular production workload or whether the ranking predicts outcomes there.

First watch condition: Watch for teams publishing evaluations that compare the index's shortlist with results on their own labeled workloads and error-cost profiles.

If those comparisons repeatedly show meaningful ranking reversals, the index should be treated as a broad screening signal rather than a deployment proxy. If rankings remain stable across task-specific tests, that would strengthen the case for using it as an efficient first-stage filter.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)