---
type: "post"
id: "artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de"
slug: "artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de"
title: "Artificial Analysis ships Endpoint Accuracy Index v1.1 with tighter long-context stats and six open-weight models"
description: "Artificial Analysis has updated its Endpoint Accuracy Index to v1.1, sharpening its long-context measurements and expanding coverage to six open-weight models."
retrieval_nugget: "Artificial Analysis has updated its Endpoint Accuracy Index to v1.1, sharpening its long-context measurements and expanding coverage to six open-weight models. The index compares provider-hosted endpoints with self-hosted references, giving teams a way to examine accuracy differences that price and throughput figures alone do not show."
published_at: "2026-11-06"
updated_at: "2026-11-06"
record_date: "2026-11-06"
date_kind: "published_at"
topics: ["agent-economics"]
entities: ["Artificial Analysis","Endpoint Accuracy Index","GLM-5.3 Flash","Meta Muse Spark Open","Berkeley Function Calling Leaderboard","Humanity's Last Exam"]
related_patterns: ["agent-economics-moves-to-completed-work"]
novelty_label: "new_fact"
what_changed: "Artificial Analysis has released Endpoint Accuracy Index v1.1, a distinct index measuring provider endpoint performance against a self-hosted baseline; prior coverage concerned its Intelligence and Capability Indexes."
prior_refs: ["https://newruntime.com/posts/artificial-analysis-launches-intelligence-index-v5-with-60-59ecd7/"]
source_url: "https://artificialanalysis.ai/methodology/endpoint-accuracy-index"
source_urls: ["https://artificialanalysis.ai/methodology/endpoint-accuracy-index"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de/","markdown":"https://newruntime.com/posts/artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de.md","json":"https://newruntime.com/posts/artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de.json"}
markdown_url: "https://newruntime.com/posts/artificial-analysis-ships-endpoint-accuracy-index-v1-1-7ed4de.md"
---

# Artificial Analysis ships Endpoint Accuracy Index v1.1 with tighter long-context stats and six open-weight models

## Retrieval answer

Artificial Analysis has updated its Endpoint Accuracy Index to v1.1, sharpening its long-context measurements and expanding coverage to six open-weight models. The index compares provider-hosted endpoints with self-hosted references, giving teams a way to examine accuracy differences that price and throughput figures alone do not show.

## Summary
Artificial Analysis has updated its Endpoint Accuracy Index to v1.1, sharpening its long-context measurements and expanding coverage to six open-weight models. The index compares provider-hosted endpoints with self-hosted references, giving teams a way to examine accuracy differences that price and throughput figures alone do not show.

## What happened
Artificial Analysis describes the Endpoint Accuracy Index as a comparison of how a specific model performs across API providers. The same model can behave differently depending on provider-side choices such as quantization, sampling defaults, context handling, token limits, prompt parsers and other configuration. The index reruns a fixed evaluation suite against each provider endpoint and expresses its overall score as a percentage of a self-hosted deployment of that model. A score of 100% means the endpoint matches the reference; a lower score indicates lower accuracy relative to it.

Version 1.1 retains three equally weighted evaluations. BFCL v4-500 tests tool and function calling on a 500-task subset of the Berkeley Function Calling Leaderboard, with three repeats. HLE-250 uses a 250-question text-only subset of Humanity’s Last Exam, with ten repeats. AA-LCR-25 measures recall and reasoning across long documents using 25 questions, with 25 repeats. Each evaluation contributes 33% to the composite. The index is only computed when all three component results are available.

The BFCL subset emphasizes harder parts of function calling, including multi-turn and parallel categories. HLE-250 is a text-only subset weighted toward more challenging questions. AA-LCR-25 asks models to reason across multiple long documents, at around 100,000 input tokens per question. The methodology identifies several ways endpoint behavior can depress these scores: tool-schema parsing and return formatting for function use; context and output-token limits, reasoning-effort limits, quantization, or inference precision for hard reasoning; and context-window limits, input truncation, or empty responses for long-context work.

The change highlighted for v1.1 is an update to AA-LCR-25, which now uses dataset version 1.1.1 and 25 repeats rather than 10. Artificial Analysis says that this narrowed the average 95% confidence interval from about ±4.7 to ±2.6 points. The suite also expands coverage to six open-weight models, including GLM-5.3 Flash and Meta Muse Spark Open. A supplied summary of the release reports more than 100 endpoints assessed and says nearly one in four scored below its self-hosted reference.

The methodology reports 95% confidence intervals based on variation across repeats, and flags whether an endpoint is statistically within the reference or significantly below it. Results are point-in-time snapshots, not live monitoring; each is dated and includes its repeat count. The index also records notes about endpoint characteristics or shortcomings that affected a result. The supplied material does not state a price for using the index or provide a date for this update.

## What the archive adds
This is a new update on a known Artificial Analysis thread. Our earlier [Intelligence Index v5 coverage](https://newruntime.com/posts/artificial-analysis-launches-intelligence-index-v5-with-60-59ecd7/) described a benchmark release that changed scoring and introduced a Progress Index; v1.1 here instead compares provider endpoints with self-hosted references. Earlier Telegram coverage also discussed Artificial Analysis’s Openness Index, but the supplied archive provides no link for that post.

## Trend signal
This update supports the practical importance of measuring agent capability at the task level rather than treating model identity as a sufficient proxy. The connection to [agent economics moves to completed work](https://newruntime.com/patterns/agent-economics-moves-to-completed-work/) is indirect: this index measures accuracy across tasks, not cost per verified completion. It reinforces that endpoint configuration can affect delivered capability, but does not establish the economics of completed work.

For a team, the useful implication is operational rather than universal. An endpoint can preserve a model’s advertised identity while altering the conditions under which the model completes a tool call, retains long input, or returns a full reasoning response. The benchmark makes those possible delivery differences visible under a fixed suite, but it does not tell a buyer which endpoint is optimal for every workload.

## Why it matters
Teams selecting an API endpoint for an open-weight model often compare price, latency and throughput. This index adds an explicit accuracy comparison against self-hosting the same model, helping expose trade-offs caused by serving choices rather than model weights alone. That can matter in tool-using and long-context workflows, where parsing, context limits or output truncation may cause a seemingly familiar model to perform differently in production.

The results should be treated as a screening signal, not a substitute for workload-specific evaluation. A team can use the reported tests to identify providers worth investigating, then rerun its own representative tasks under its own prompts, data and success criteria. The confidence intervals and point-in-time labels are particularly important: small score differences may not be meaningful, and endpoint behavior can change after a snapshot.

A practical procurement consequence follows from the design of the suite. Contract terms that list a model name, token price, and nominal context window do not by themselves demonstrate comparable task performance. Teams with high-stakes agent workflows can add a small endpoint acceptance test: run the same tool schemas, long inputs, and completion constraints against candidate providers, record failures and truncations, and repeat after major serving changes. The index offers an independent starting structure for that evaluation.

## Limits and uncertainty
The index covers only three selected evaluations and fixed subsets: it does not establish how endpoints perform across every task, language, prompt style or production workload. Two component scores rely on an equality-checker language model; the tool-calling score uses execution and matching. The source says results are point-in-time snapshots, not live monitoring, so they cannot guarantee current endpoint behavior.

Confidence intervals reflect variation across repeats, not every source of uncertainty. A self-hosted deployment is the reference baseline in this design, rather than proof of a model’s absolute capability. The supplied material also does not include the underlying endpoint-by-endpoint results, pricing, or details of how the six-model expansion was selected. It therefore supports comparison and investigation, not a general claim that any particular provider is unsuitable.

## Skill to build now
Create a provider-parity test for one important agent workflow this week. Use a fixed prompt, the same tool schema, a realistic long input, and an explicit success condition; then run it through each candidate endpoint and retain both raw outputs and failure reasons. Track truncation, malformed tool calls, empty responses, and task completion separately from latency and token cost. Repeat the test after provider configuration changes rather than assuming a model label guarantees stable behavior.

## Reader takeaway
A model name does not guarantee the same capability across providers. Artificial Analysis v1.1 offers a more precise, though still bounded, way to compare API endpoints with self-hosted references. Use it to shortlist candidates, then validate the endpoints on your own tasks before choosing based on cost or speed alone.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agent Economics](https://newruntime.com/topics/agent-economics.md): Explore the agent-economics topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)