Epoch AI introduced Benchmark Reviews, a new initiative to audit AI benchmarks. It launched with 15 benchmarks: 4 rated Verified, 9 rated Flawed, and 2 where there was not enough information for a review. Ethan Mollick described Epoch as doing some of the best public benchmarking work on AI and said the results show how bad the state of benchmarking is.
Auditing benchmarks directly exposes that many widely cited AI evaluations may not measure what their scores are assumed to measure. Epoch sorts each benchmark it reviews into Verified, Flawed, or insufficient information, which turns a single leaderboard number into a judgment about whether the test design itself can be trusted. Nine of fifteen benchmarks falling into the Flawed category means that the majority of the initial sample did not pass the audit.
Model releases continue to lean on benchmark scores, so a public audit that labels most reviewed benchmarks as flawed changes how those scores should be read. Engineers choosing models from published scores should check whether the cited benchmarks have been reviewed and how they were classified.
The announcement gives only aggregate counts and does not, in the fetched text, name which benchmarks were judged flawed or explain the specific defects. The list of named benchmarks and the stated reasons for each Flawed verdict will show which common evaluations are most affected. Whether model developers stop citing benchmarks rated Flawed in release materials would indicate if the audit changes reporting practice.