---
type: "post"
id: "we-don-t-review-our-own-benchmarks-instead-e4fd67"
slug: "we-don-t-review-our-own-benchmarks-instead-e4fd67"
title: "We don’t review our own benchmarks. Instead, we’ve contracted with Generality Labs to independently audit them."
description: "Epoch AI says Generality Labs, an independent benchmark-auditing organization, audited its Chess Puzzles benchmark in October 2026 and issued a Verified verdict."
retrieval_nugget: "Epoch AI says Generality Labs, an independent benchmark-auditing organization, audited its Chess Puzzles benchmark in October 2026 and issued a Verified verdict."
published_at: "2026-10-24"
updated_at: "2026-10-24"
record_date: "2026-10-24"
date_kind: "published_at"
topics: ["evals","verification"]
entities: ["Epoch AI","Generality Labs","Chess Puzzles","EBR-bench","Stockfish"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "Epoch says Generality Labs will independently audit its benchmarks, including the Chess Puzzles benchmark; the prior covered Epoch’s automation research, not this external review arrangement."
prior_refs: ["memory:a187128b-d516-8c3e-a423-2647e34f7bab"]
source_url: "https://epoch.ai/benchmarks/chess-puzzles?tab=release-date&view=graph"
source_urls: ["https://epoch.ai/benchmarks/chess-puzzles?tab=release-date&view=graph"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:we-don-t-review-our-own-benchmarks-instead-e4fd67"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/we-don-t-review-our-own-benchmarks-instead-e4fd67/","markdown":"https://newruntime.com/posts/we-don-t-review-our-own-benchmarks-instead-e4fd67.md","json":"https://newruntime.com/posts/we-don-t-review-our-own-benchmarks-instead-e4fd67.json"}
---

# We don’t review our own benchmarks. Instead, we’ve contracted with Generality Labs to independently audit them.

## Retrieval answer

Epoch AI says Generality Labs, an independent benchmark-auditing organization, audited its Chess Puzzles benchmark in October 2026 and issued a Verified verdict.

Epoch AI says Generality Labs, an independent benchmark-auditing organization, audited its Chess Puzzles benchmark in October 2026 and issued a Verified verdict. Epoch says it contracted Generality Labs rather than reviewing its own benchmarks.

Chess Puzzles contains 100 programmatically generated chess positions. Models receive a FEN board state and must submit the single best next move, judged against Stockfish. Epoch released version 2 on October 14, 2026 and says it re-ran every model it could. The source says version 2.0.0 uses a `qa_agent` scaffold to prompt a model that stops before submitting, report remaining tokens, and provide a submission tool; Epoch says this fixes major issues identified in the audit.

The source also says Generality Labs had audited EBR-bench, but the supplied Chess Puzzles page documents the audit details and response for Chess Puzzles. It does not establish the auditors' full methodology, the terms of Epoch's contract, or whether an audit verifies broader claims about model reasoning.

Source: https://epoch.ai/benchmarks/chess-puzzles?tab=release-date&view=graph

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Verification](https://newruntime.com/topics/verification.md): Explore the verification topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)