---
type: "post"
slug: "a-rosetta-stone-for-ai-benchmarks"
title: "A Rosetta Stone for AI benchmarks"
description: "Epoch AI stitches saturating benchmarks into one scale with a statistical framework, because any single benchmark only separates models that are neither too bad nor too good."
retrieval_nugget: "Epoch AI stitches saturating benchmarks into one scale with a statistical framework, because any single benchmark only separates models that are neither too bad nor too good."
published_at: "2026-09-02"
updated_at: "2026-09-12"
record_date: "2026-08-25"
date_kind: "discovered_at"
topics: ["agents","ai","evaluation"]
entities: ["Epoch AI"]
source_url: "https://epoch.ai/publications/a-rosetta-stone-for-ai-benchmarks"
source_title: "A Rosetta Stone for AI benchmarks"
source_domain: "epoch.ai"
source_terms: ["Rosetta","Stone","benchmarks","benchmark","model"]
summary_word_count: 190
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:a-rosetta-stone-for-ai-benchmarks"
status: "published"
source_urls: ["https://epoch.ai/publications/a-rosetta-stone-for-ai-benchmarks"]
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:68d66b74548effc4c0a30164c14d164d17021acc9b37e3bf6f58882cdbf66c00","reviewed_at":"2026-09-12T10:00:00Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":1,"watch_condition_count":1,"related_record_count":3}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Epoch AI published a paper proposing a statistical framework that stitches benchmarks together into a common scale, aimed at a structural problem: most benchmarks saturate too quickly to study long-run trends in AI capability.","observed_facts":[{"text":"The paper states that most benchmarks saturate too quickly to study long-run AI trends.","source_urls":["https://epoch.ai/publications/a-rosetta-stone-for-ai-benchmarks"]},{"text":"It proposes a statistical framework that stitches benchmarks together, with implications for algorithmic progress and forecasting.","source_urls":["https://epoch.ai/publications/a-rosetta-stone-for-ai-benchmarks"]}],"mechanism":"A benchmark therefore only separates models in its middle range, and since models improve quickly, each benchmark provides a usable signal for a short window before it stops measuring anything.","why_now":"A really bad model scores 0% on a benchmark — but so does an extremely bad one, so the benchmark cannot distinguish them.","implications":["A stitched scale inherits every weakness of the benchmarks it joins."],"evidence_boundary":"This is a paper with published source code rather than a settled method, and the stitched trend depends on modelling assumptions about how benchmarks overlap.","watch_conditions":["The condition to watch is whether independent groups reproduce the same long-run curve from different benchmark sets, because a scale that only one framework can compute is a forecast rather than a measurement."],"related_records":[{"url":"https://newruntime.com/posts/evals-move-from-leaderboards-to-operating-diagnostics","relation":"Prior coverage of evaluation as an operating diagnostic."},{"url":"https://newruntime.com/posts/deepmind-double-blind-frontier-model-evaluations","relation":"Prior coverage of evaluation integrity conditions."},{"url":"https://newruntime.com/posts/minimum-integrity-stack-agent-leaderboard","relation":"Prior coverage of the floor for quoting a leaderboard."}]}
routes: {"html":"https://newruntime.com/posts/a-rosetta-stone-for-ai-benchmarks/","markdown":"https://newruntime.com/posts/a-rosetta-stone-for-ai-benchmarks.md","json":"https://newruntime.com/posts/a-rosetta-stone-for-ai-benchmarks.json"}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
---

# A Rosetta Stone for AI benchmarks

## Retrieval answer

Epoch AI stitches saturating benchmarks into one scale with a statistical framework, because any single benchmark only separates models that are neither too bad nor too good.

Epoch AI published a paper proposing a statistical framework that stitches benchmarks together into a common scale, aimed at a structural problem: most benchmarks saturate too quickly to study long-run trends in AI capability.

The reasoning is simple enough to be uncomfortable. A really bad model scores 0% on a benchmark — but so does an extremely bad one, so the benchmark cannot distinguish them. A really good model scores 100%, and so does an extremely good one. A benchmark therefore only separates models in its middle range, and since models improve quickly, each benchmark provides a usable signal for a short window before it stops measuring anything. Stitching overlapping benchmarks lets that narrow window be extended into a longer curve, with consequences the authors tie to estimates of algorithmic progress and to forecasting.

Treating the measurement instrument as the object of study is the move behind [evals moving from leaderboards to operating diagnostics](https://newruntime.com/posts/evals-move-from-leaderboards-to-operating-diagnostics/), and it depends on the integrity conditions in [double-blind frontier evaluations](https://newruntime.com/posts/deepmind-double-blind-frontier-model-evaluations/) and [the minimum integrity stack for an agent leaderboard](https://newruntime.com/posts/minimum-integrity-stack-agent-leaderboard/). A stitched scale inherits every weakness of the benchmarks it joins.

This is a paper with published source code rather than a settled method, and the stitched trend depends on modelling assumptions about how benchmarks overlap. The condition to watch is whether independent groups reproduce the same long-run curve from different benchmark sets, because a scale that only one framework can compute is a forecast rather than a measurement.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Ai](https://newruntime.com/topics/ai.md): Explore the ai topic hub.
- topic hub: [Evaluation](https://newruntime.com/topics/evaluation.md): Explore the evaluation topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)