{"id":"nr-epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe","slug":"epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe","title":"Epoch AI launches Benchmark Reviews, rating 9 of 15 benchmarks flawed","observed_at":"2026-09-18T13:48:49+00:00","record_date":"2026-09-18","date_kind":"observed_at","novelty":"new","source_platform":"x.com","topics":["benchmarks","evaluation"],"entities":["Epoch AI"],"related_patterns":[],"source_url":"https://x.com/emollick/status/2100744220562673683","source_urls":["https://x.com/emollick/status/2100744220562673683"],"basket":{"id":"0325e6b8-f527-4f9b-bfa3-3f30e57f380b","revision":1,"cluster_id":"fdd6fe57-a058-415e-8b52-9f4badb92bb1"},"type":"post","description":"Epoch AI began auditing AI benchmarks, launching with 15 reviews: 4 Verified, 9 Flawed and 2 lacking enough information for a verdict.","why_it_matters":"The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects. The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected. Whether model developers stop citing benchmarks rated Flawed in release materials would indicate if the audit changes reporting practice.","verification_level":"source-inspected","signal_type":"field_note","schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe","retrieval_nugget":"Epoch AI began auditing AI benchmarks, launching with 15 reviews: 4 Verified, 9 Flawed and 2 lacking enough information for a verdict. Epoch AI introduced Benchmark Reviews, a new initiative to audit AI benchmarks. It launched with 15 benchmarks: 4 rated Verified, 9 rated Flawed, and 2 where there was not enough information for a review. Ethan Mollick described Epoch","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:988f4faa97e60d8acb8c4784e03ca4e64ae5c6b5bc598ab6443c6a62e290dc61","reviewed_at":"2026-09-24T13:28:46Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":1,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"Auditing benchmarks directly exposes that many widely cited AI evaluations may not measure what their scores are assumed to measure.","observed_facts":[{"text":"Epoch AI introduced Benchmark Reviews, an initiative to audit AI benchmarks.","source_urls":["https://x.com/emollick/status/2100744220562673683"]},{"text":"The initiative launched with reviews of 15 benchmarks.","source_urls":["https://x.com/emollick/status/2100744220562673683"]},{"text":"Four benchmarks were rated Verified, nine Flawed, and two lacked enough information for review.","source_urls":["https://x.com/emollick/status/2100744220562673683"]}],"mechanism":"Epoch sorts each benchmark it reviews into Verified, Flawed, or insufficient information, which turns a single leaderboard number into a judgment about whether the test design itself can be trusted.","why_now":"Model releases continue to lean on benchmark scores, so a public audit that labels most reviewed benchmarks as flawed changes how those scores should be read.","implications":["Engineers choosing models from published scores should check whether the cited benchmarks have been reviewed and how they were classified."],"evidence_boundary":"The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects.","watch_conditions":["The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected."],"related_records":[],"new_branch_reason":"A systematic public audit of benchmark validity is a new evaluation resource that affects how all later model claims are read."},"routes":{"html":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe/","markdown":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe.md","json":"https://newruntime.com/posts/epoch-ai-launches-benchmark-reviews-rating-9-of-15-fdd6fe.json"},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"next_reads":[{"type":"topic","path":"/topics/benchmarks/","url":"https://newruntime.com/topics/benchmarks/","title":"Benchmarks","reason":"Explore the benchmarks topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/benchmarks.md","json_url":"https://newruntime.com/topics/benchmarks.json"},{"type":"topic","path":"/topics/evaluation/","url":"https://newruntime.com/topics/evaluation/","title":"Evaluation","reason":"Explore the evaluation topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evaluation.md","json_url":"https://newruntime.com/topics/evaluation.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}