Field noteOct 24, 20261 min read1 source

We recently launched Benchmark Reviews , an initiative led by senior researcher Michelle Campeau, where we audit external AI benchmarks and assess whether their scores accurately measure what they...

In one minute

Epoch AI has launched Benchmark Reviews, an initiative for independent reviews of external AI benchmarks.

What changed: Epoch has launched its own independent reviews of external AI benchmarks, using verdicts to indicate whether scores are interpretable, flawed, or insufficiently documented.

Its documentation says the reviews assess whether benchmark scores measure what they are described as measuring.

The program assigns one of four verdicts. “Verified” means a benchmark can broadly be interpreted as described and any errors do not substantially affect results. “Flawed” denotes one or more substantive issues relevant to interpretation. “Not enough info” means Epoch lacks sufficient material to review it. Benchmarks created by Epoch will receive a “By Epoch” label and will not be reviewed because of a conflict of interest.

Epoch invites suggestions for benchmarks at reviews@epoch.ai. The published documentation does not establish which specific benchmarks have been reviewed, how many have received each verdict, or the results of any individual audit.

Evidence / sources

  1. 1epoch.ai/data/benchmark-reviews-documentation
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicEvalsExplore the evals topic hub.
  2. 02topicVerificationExplore the verification topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04source ledgerSource LedgerInspect the public source evidence graph.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract