The flagged item is a public audit, not a single paper: the ICML 2026 Open Reproductions challenge, run by Hugging Face and alphaXiv from 15 July through 2 August 2026. Community members pointed coding agents at accepted papers, published Trackio logbooks (static Hugging Face Spaces with write-ups, code, artifacts, and optional traces), and submitted them to an automated Logbook Judge running GLM-5.2. Hugging Face’s 13 August 2026 report is the results write-up; the challenge FAQ, agent prompt, organization page, and frozen verdicts dataset are the operational record.
This matters because ICML 2026 was large enough that volunteer review could not inspect every proof and run. Hugging Face cites 23,918 submissions and 6,352 accepted papers; ICML’s own fact sheet lists 24,661 main-track submissions and 6,552 accepted. ICML’s program chairs had already warned that generative tools were straining review. Agent reproduction is a proposed second layer: executable checks of named claims, not a replacement for judging novelty.
What we checked
- The Hugging Face and alphaXiv ICML 2026 Open Reproductions challenge used coding agents to attempt claim-level reproduction of a large share of accepted papers and published measured success rates. confirmed. From 15 July to 2 August 2026, 1,221 participants published 6,816 Trackio logbooks covering 2,226 papers, about 34 percent of the conference on Hugging Face’s count, with 35,908 claims judged. Among examined papers, 51 percent (1,103) had at least one independently verified claim, including 266 fully reproduced, and 23 percent (496) had at least one falsified or contested claim. [1][6][3]
- The challenge used defined claim-level evaluation criteria and documented failure modes rather than a single reproduce-or-not score. confirmed. The agent prompt and FAQ define four Logbook Judge verdicts—verified, falsified, toy, and inconclusive—worth 2, 2, 1, and 0 points, with logbook self-assessments treated as untrusted. Published failure modes include 502 papers with toy-scale evidence only, 280 inconclusive (missing artifacts most often), 242 papers where independent teams reached opposite verdicts, short-horizon false verifies, and at least one units-mismatch false falsification. [4][5][1]
What we found
- Hugging Face reports 6,816 logbooks, 2,226 papers attempted after indexing 6,341 accepted papers, 35,908 judged claims, 2,962 Hugging Face Jobs, and 274 public agent-trace datasets. Paper-level outcomes: 266 fully reproduced; 632 partially reproduced with nothing falsified; 3,978 claims confirmed with real experiments; 496 papers with at least one falsified or contested claim, including 49 with all claims falsified; 502 toy-scale only; 280 inconclusive. Those bins are the organizers’ figures; they do not sum cleanly to 2,226 in the published text, so they should not be treated as a fully reconciled census. [1][2][3]
- The task was claim-level, not “rerun the paper.” Organizers extracted core claims so agents had checkable targets. Participants used Claude Code, Codex, Cursor, OpenResearch’s orx, and others. When official code, data, or checkpoints were missing, the prompt required an independent scaffold, including synthetic proxies, labeled as toy. Scoring paid full credit for a documented full falsification, so a negative result was not a consolation prize. [4][5][6]
- The judge is an LLM. Hugging Face says GLM-5.2 re-read every logbook and ignored self-grades. The model card for zai-org/GLM-5.2 describes an MIT-licensed open-weights model with a 1M-token context; that confirms the named judge exists, not that its verdicts are reliable. A participant report states the judge scored only the official anchored claim list, typically three to six claims per paper, under a 120,000-character reading window, and that an identical logbook once flipped from verified to inconclusive after a re-sync. [1][7][8]
- Organizers did not take agent falsifications at face value. Thirty-five participants formally claimed a falsification; Hugging Face says it re-read the paper and logbook and re-derived the math or re-ran the experiment. Confirmed examples include a spotlight paging paper whose claimed O(1) additive robustness grew like log k, a Frank-Wolfe attention theorem with exact-arithmetic counterexamples after hundreds of steps, a theory/code reverse-KL versus forward-KL mismatch, and an evaluation diluted by EOS padding. One claimed “2x slower than baseline” was a per-trajectory versus per-batch-of-50 units bug; the same data supported the paper’s 8x speedup. [1][3]
- Independent reporting and a second study bound the claim. Science, citing organizers, says at least 12 papers were confirmed to contain real errors missed by reviewers, and that winner Jansen Tang’s agents covered 363 papers. Separately, SAI reviewed all 168 ICML 2026 oral papers and completed 105 full replications: among 92 with at least five attempted claims, only 8 reproduced more than 80 percent of tested claims and 34 more than 40 percent. SAI’s median estimated cost to reproduce an oral paper’s reported experiments was about $8,900; 17 of 105 exceeded $100,000; the highest estimate was nearly $2.2 million. [3][9]
- Compute and artifacts were binding constraints. Participants received $20 in Hugging Face credits; 750 GPU-credit slots filled by 17 July. The FAQ allows a similar-class open-model swap when the backbone is not the contribution, while reserving toy for reduced scale. SAI found 104 of 168 orals shipped code it could run, and four papers depended on retired models. ICML’s fact sheet and Hugging Face’s submission counts disagree, so “34 percent of the conference” is Hugging Face’s fraction of its indexed set, not a number ICML itself published for this audit. [6][5][9][10]
What it means
Treat agent reproduction as an evidence pipeline, not a truth bit. The useful design is explicit claims, command-and-artifact logbooks, a judge that ignores self-grades, independent reruns, and humans at disagreement, scale-dependent proofs, perceptual quality, and unit conversions. Score honest falsification. Do not ship a “verified” badge from a short-horizon or toy run. Missing artifacts are the modal inconclusive path, so artifact access is part of the product.
Do not read 266 fully reproduced papers as “the conference is fine,” or 496 falsified-or-contested papers as “a quarter of ICML is wrong.” Coverage was participant-chosen, many checks were reduced-scale, and the grader is an LLM. SAI’s oral-paper numbers are harsher and show why full-scale reruns stay rare unless venues require runnable artifacts or fund them. The build target is a public claim-level trail people can audit, not an autonomous accept/reject.
Still open
- How many of the 496 falsified-or-contested papers survive author response and human re-derivation beyond the “at least 12” Science attributed to organizers, and how many of the 35 claimed falsifications were confirmed versus withdrawn?
- Were the 2,226 attempted papers a representative slice of ICML 2026, or biased toward papers with code, theory that fits a laptop, or leaderboard-friendly claim lists?
- How stable is GLM-5.2 as a judge across re-syncs, reading-window truncation, and mid-challenge claim-list changes, and is there a published inter-annotator study against humans?
- Which official conference size should audits use, given Hugging Face’s 23,918/6,352 versus ICML’s fact-sheet 24,661/6,552?