---
id: "nr-epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe"
slug: "epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe"
title: "Epoch AI launches Benchmark Reviews, rating 9 of 15 benchmarks flawed"
observed_at: "2026-09-18T13:48:49+00:00"
record_date: "2026-09-18"
date_kind: "observed_at"
novelty: "new"
source_platform: "x.com"
topics: ["benchmarks","evaluation"]
entities: ["Epoch AI"]
related_patterns: []
source_url: "https://x.com/emollick/status/2100744220562673683"
source_urls: ["https://x.com/emollick/status/2100744220562673683"]
basket: {"id":"0325e6b8-f527-4f9b-bfa3-3f30e57f380b","revision":1,"cluster_id":"fdd6fe57-a058-415e-8b52-9f4badb92bb1"}
type: "post"
description: "Epoch AI began auditing AI benchmarks, launching with 15 reviews: 4 Verified, 9 Flawed and 2 lacking enough information for a verdict."
why_it_matters: "The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects. The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected. Whether model developers stop citing benchmarks rated Flawed in release materials would indicate if the audit changes reporting practice."
verification_level: "source-inspected"
signal_type: "field_note"
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe"
retrieval_nugget: "Epoch AI began auditing AI benchmarks, launching with 15 reviews: 4 Verified, 9 Flawed and 2 lacking enough information for a verdict. Epoch AI introduced Benchmark Reviews, a new initiative to audit AI benchmarks. It launched with 15 benchmarks: 4 rated Verified, 9 rated Flawed, and 2 where there was not enough information for a review. Ethan Mollick described Epoch"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:988f4faa97e60d8acb8c4784e03ca4e64ae5c6b5bc598ab6443c6a62e290dc61","reviewed_at":"2026-09-24T13:28:46Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Auditing benchmarks directly exposes that many widely cited AI evaluations may not measure what their scores are assumed to measure.","observed_facts":[{"text":"Epoch AI introduced Benchmark Reviews, an initiative to audit AI benchmarks.","source_urls":["https://x.com/emollick/status/2100744220562673683"]},{"text":"The initiative launched with reviews of 15 benchmarks.","source_urls":["https://x.com/emollick/status/2100744220562673683"]},{"text":"Four benchmarks were rated Verified, nine Flawed, and two lacked enough information for review.","source_urls":["https://x.com/emollick/status/2100744220562673683"]}],"mechanism":"Epoch sorts each benchmark it reviews into Verified, Flawed, or insufficient information, which turns a single leaderboard number into a judgment about whether the test design itself can be trusted.","why_now":"Model releases continue to lean on benchmark scores, so a public audit that labels most reviewed benchmarks as flawed changes how those scores should be read.","implications":["Engineers choosing models from published scores should check whether the cited benchmarks have been reviewed and how they were classified."],"evidence_boundary":"The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects.","watch_conditions":["The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected."],"related_records":[],"new_branch_reason":"A systematic public audit of benchmark validity is a new evaluation resource that affects how all later model claims are read."}
routes: {"html":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe/","markdown":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe.md","json":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe.json"}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
---

# Epoch AI launches Benchmark Reviews, rating 9 of 15 benchmarks flawed

## Retrieval answer

Epoch AI began auditing AI benchmarks, launching with 15 reviews: 4 Verified, 9 Flawed and 2 lacking enough information for a verdict. Epoch AI introduced Benchmark Reviews, a new initiative to audit AI benchmarks. It launched with 15 benchmarks: 4 rated Verified, 9 rated Flawed, and 2 where there was not enough information for a review. Ethan Mollick described Epoch

Epoch AI introduced Benchmark Reviews, a new initiative to audit AI benchmarks. It launched with 15 benchmarks: 4 rated Verified, 9 rated Flawed, and 2 where there was not enough information for a review. Ethan Mollick described Epoch as doing some of the best public benchmarking work on AI and said the results show how bad the state of benchmarking is.

Auditing benchmarks directly exposes that many widely cited AI evaluations may not measure what their scores are assumed to measure. Epoch sorts each benchmark it reviews into Verified, Flawed, or insufficient information, which turns a single leaderboard number into a judgment about whether the test design itself can be trusted. Nine of fifteen benchmarks falling into the Flawed category means that the majority of the initial sample did not pass the audit.

Model releases continue to lean on benchmark scores, so a public audit that labels most reviewed benchmarks as flawed changes how those scores should be read. Engineers choosing models from published scores should check whether the cited benchmarks have been reviewed and how they were classified.

The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects. The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected. Whether model developers stop citing benchmarks rated Flawed in release materials would indicate if the audit changes reporting practice.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Benchmarks](https://newruntime.com/topics/benchmarks.md): Explore the benchmarks topic hub.
- topic hub: [Evaluation](https://newruntime.com/topics/evaluation.md): Explore the evaluation topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)