---
type: "post"
id: "openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554"
slug: "openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554"
title: "OpenFrontiers benchmark finds LLM agents rarely rediscover recently solved math problems without web access"
description: "OpenFrontiers pairs a corpus of 12,151 open research problems with a benchmark of 200 mathematical problems, then tests seven LLM agents on a curated subset."
retrieval_nugget: "OpenFrontiers pairs a corpus of 12,151 open research problems with a benchmark of 200 mathematical problems, then tests seven LLM agents on a curated subset. Without web access, the agents rarely rediscovered recently solved problems; with live web access, success mostly reflected retrieval of published solutions. That distinction matters when evaluating research agents."
published_at: "2026-11-01"
updated_at: "2026-11-01"
record_date: "2026-11-01"
date_kind: "published_at"
topics: ["agents","evals"]
entities: ["OpenFrontiers","Jiarui Yao","Ruida Wang","Tong Zhang","Lean 4"]
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
novelty_label: "new_fact"
what_changed: "OpenFrontiers finds closed-book rediscovery rare—at most 7 of 40 resolved problems—and says live-web success mostly reflects retrieval, adding a math-specific result to prior AI research-problem benchmarks."
prior_refs: ["https://newruntime.com/posts/a-team-of-35-researchers-put-together-50-1f091c/"]
source_url: "https://arxiv.org/abs/2610.37739"
source_urls: ["https://arxiv.org/abs/2610.37739"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554/","markdown":"https://newruntime.com/posts/openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554.md","json":"https://newruntime.com/posts/openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554.json"}
markdown_url: "https://newruntime.com/posts/openfrontiers-benchmark-finds-llm-agents-rarely-rediscover-recently-a83554.md"
---

# OpenFrontiers benchmark finds LLM agents rarely rediscover recently solved math problems without web access

## Retrieval answer

OpenFrontiers pairs a corpus of 12,151 open research problems with a benchmark of 200 mathematical problems, then tests seven LLM agents on a curated subset. Without web access, the agents rarely rediscovered recently solved problems; with live web access, success mostly reflected retrieval of published solutions. That distinction matters when evaluating research agents.

## Summary
OpenFrontiers pairs a corpus of 12,151 open research problems with a benchmark of 200 mathematical problems, then tests seven LLM agents on a curated subset. Without web access, the agents rarely rediscovered recently solved problems; with live web access, success mostly reflected retrieval of published solutions. That distinction matters when evaluating research agents.

## What happened
Jiarui Yao, Ruida Wang and Tong Zhang introduced OpenFrontiers in a paper submitted to arXiv on 27 October 2026. The project combines two related resources. The first is a structured corpus of 12,151 open research problems. According to the paper, its records include problem statements, context, progress and other information. The second is a retrospective benchmark of mathematical research problems intended to test scientific rediscovery.

The benchmark contains 200 problems. Of these, 110 were resolved in 2026 and 90 are still open. The resolved problems create a useful retrospective test: they were research problems, but the evaluators have documented solutions against which to assess an agent's work. The paper frames this as a way to move beyond a collection of general research-style questions toward a set where claims about a proposed proof can be checked.

Each benchmark item has a Lean 4 statement. The authors say the statements were checked by automated probes and two independent reviews. Lean 4 is used here as a formal representation for checking proof-oriented outputs, rather than relying only on a grader's reading of prose. OpenFrontiers also defines a filtered HQ-50 subset, selected on a quality score and other criteria. The supplied abstract does not provide the complete filtering rule, but it is this subset on which the reported agent evaluation is run.

The evaluation protocol grades formal proofs using two independent checkers in an isolated sandbox. It compares three information settings: closed-book, frozen retrieval and live web. That design matters because a system can produce a correct proof through different routes: deriving an answer from its own available knowledge, consulting a fixed retrieval collection, or locating a published solution online. A correctness-only result would blur those routes.

The authors evaluated seven LLM agents on HQ-50 under a standardized budget. In the closed-book setting, rediscovery was rare: the strongest reported outcome was at most 7 of 40 resolved problems. In the live-web condition, the paper says success mostly reflected retrieval of published solutions, rather than independent derivation. The abstract identifies frozen retrieval as a test condition but does not report its result, and it does not identify the individual agents or their scores.

OpenFrontiers therefore separates two propositions often collapsed in claims about research agents. One is that a system can return a formally valid proof. The other is that it can independently rediscover a result that was once unknown. The corpus and benchmark link broad problem collection to a formally graded subset with documented solutions. The authors present the resource as a foundation for measuring research-level rediscovery and for studying progress toward solving still-open scientific problems. The supplied source does not specify a price, licence, or access restrictions for the dataset and benchmark.

## What the archive adds
Our [Continual Learning Exam coverage](https://newruntime.com/posts/continual-learning-exam-benchmark-finds-best-llm-agent-44b039/) covered a different five-level cumulative-learning benchmark, reporting that its best agent completed 47.7% of 600 progressive professional-task levels. OpenFrontiers is a new fact on the broader question of what an agent benchmark measures: its central distinction is independent rediscovery versus retrieval of an already published solution.

## Trend signal
This supports the need for inspectable evidence when evaluating agent output: formal statements, review, and independent proof checking make the result easier to audit than a fluent answer alone. But no supplied SITE_PATTERNS thesis specifically addresses research-problem rediscovery or benchmark contamination, so no pattern link is close enough to claim direct support.

## Why it matters
For teams building research workflows, a valid answer and an independently generated discovery are different products. OpenFrontiers reports that live web access mostly helped agents retrieve published solutions, while closed-book rediscovery was rare. An evaluation that reports only solved tasks can therefore overstate a system's capacity for novel research work.

The practical response is to label the information regime of every evaluation: closed-book reasoning, constrained retrieval, or open-web research. Teams should also preserve provenance for key claims and distinguish a sourced answer from a candidate result requiring novelty review. OpenFrontiers is not a complete measurement of scientific work, but its protocol offers a concrete design principle: test correctness separately from the route by which a model reached the answer.

Its use of Lean 4 statements, automated probes, independent reviews, and two proof checkers also illustrates an artifact-first approach. In high-stakes research settings, builders need more than a model narrative and a pass/fail score. They need inspectable outputs and a record of the resources that may have shaped them.

## Limits and uncertainty
The supplied abstract does not name the seven agents, give their individual scores, or state the exact standardized budget. It reports only that closed-book rediscovery reached at most 7 of 40 resolved problems and describes live-web results qualitatively. It also does not provide the frozen-retrieval result, the full HQ-50 selection criteria, or detailed procedures for identifying retrieval of published solutions.

The reported benchmark is mathematics-focused and curated. It does not establish that the same results apply to other scientific disciplines, to open-ended laboratory work, or to all forms of research reasoning. Nor does it show whether an agent can make a new discovery on a problem that remains open. The supplied material also does not specify the dataset's licence, availability terms, or price.

## Reader takeaway
A correct research answer is not, by itself, evidence of independent discovery. OpenFrontiers separates formal proof checking from the information path that produced a solution—and finds that web-enabled success mostly meant retrieving published work. Teams evaluating research agents should expose that distinction rather than treating every verified output as novel reasoning.

## Open question
How do the seven agents perform under frozen retrieval, and how much does the result change outside this curated mathematics subset?

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Agents](https://newruntime.com/topics/agents.md): Explore the agents topic hub.
- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)