What happened
Ai2 released a study, benchmark, data, code, and an automated pipeline for evaluating “novelty judges”: LLM-based systems that determine whether a research idea is new relative to prior work. The release is called Old Ideas, Novel Problems. Ai2 frames the problem as a missing counterpart to automated idea generation: a system can propose plausible directions, but scientific-discovery workflows also need a dependable way to tell whether a direction contributes something genuinely new.
The benchmark begins with ICLR 2026 peer reviews rather than treating conference acceptance as a novelty label. Ai2 mined review passages where reviewers explicitly discussed originality. Its Human-Only evaluation pool contains 154 human-authored ideas from accepted papers in the top-rated 10% of their research area, with reviewers positively identifying novelty and none disputing it. The lower-novelty pool contains 145 ideas from rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty. In a check of this review-mining process, an annotator agreed with the extracted novelty-signal polarity in 47 of 50 cases; Ai2 says none of the disagreements changed whether a review counted as positive or negative novelty evidence.
Ai2 also created a Human+Generated setting. It pairs the same 154 validated human ideas with 154 ideas from a simple LLM generator. The generated ideas are weak labels rather than individually verified novelty judgments: the generator received a research area and an instruction to propose a novel idea, without literature search, tools, iteration, or feedback. An expert reviewed 30 pairs, weighted toward cases the strongest judge got wrong, preferred the human idea in all 30, and judged none of the generated ideas novel. Ai2 also rewrote ideas into a common structure to check whether the judges were responding to stylistic differences.
Across six judges—GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6—the study varied prompts, reasoning budgets, related-paper access, idea format, aggregation of judgments, and, for generated ideas, the generator model. Prompting produced the largest changes. On the same Human+Generated pairs, a GPT-5.4 prompt saying exactly one idea was novel yielded 93.5% accuracy; a simpler prompt asking which idea was novel yielded 40.9%, below chance. The model changed its verdict on more than half the pairs. A comparable prompt change moved Claude Opus 4.6’s Human-Only accuracy by 21 points.
More capability was not automatically better. Ai2 tested GPT-5.6 Sol with an extra-high reasoning budget and live iterative search limited to arXiv. It reached 0.78 macro-F1 on Human+Generated, 11 points behind a Claude Opus 4.6 setup at 0.89, and cost roughly ten times more per decision at current API prices. A low-reasoning Claude Opus 4.6 baseline with majority voting across three calls achieved 0.854 macro-F1 at a stated total evaluation cost of $2.59. Ai2 reports that Idea Novelty Checker scored 0.642 at $30.66 and Sakana AI’s AI Scientist novelty evaluator scored 0.772 at $82.05 in its comparison.
What the archive adds
This is a new fact on a related Ai2 story. Our earlier note on LitPivot: Developing research ideas through dynamic contextualization within the literature landscape covered a literature-grounded ideation system. Old Ideas, Novel Problems addresses the separate evaluation problem: whether the judge that filters or ranks such generated ideas is stable enough to trust.
Trend signal
The release supports Scientific agents need artifact evidence. Ai2 does not claim to solve novelty evaluation; it publishes a benchmark, data, and a pipeline that make disagreement and setup sensitivity inspectable. That shifts the useful question from whether an agent can generate an idea to whether its novelty claim can be tested against an evidence-bearing evaluation process.
Connections to other events
LitPivot is the closest archive connection because both releases concern literature-grounded AI support for research ideation. The difference matters: LitPivot concerns developing candidate ideas in a literature landscape, while this benchmark tests the reliability of systems asked to evaluate whether candidate ideas are novel.
Why it matters
Teams building scientific agents often place an LLM judge at a critical routing point: rank ideas, reject duplicates, decide what receives researcher attention, or claim that one ideation system outperforms another. This release says that a judge’s output is not a portable measurement merely because the underlying model is strong. Prompt wording, the presence of retrieval, the reasoning configuration, and the form of the candidate idea can materially alter the result.
The practical consequence is that novelty evaluation needs to be treated as an evaluated subsystem, not a final API call. A workflow can look efficient while silently changing which research directions it funds or discards. Ai2’s cost comparison adds another warning: expensive search-heavy or purpose-built systems are not, in these tests, guaranteed to be more accurate than a simpler prompted baseline. Robustness evidence therefore matters alongside headline quality and cost.
What happens next
- Research-agent teams will add prompt- and model-sensitivity checks to novelty evaluations. (6 months, 65%). Watch for: benchmark reports publishing verdict agreement across setups rather than one accuracy score.
- Benchmark builders will test judges on generated ideas, not only human-authored papers. (12 months, 60%). Watch for: evaluation sets with separate human and generated-idea splits.
Limits and uncertainty
The release establishes instability in Ai2’s tested settings; it does not establish that every novelty judge is unreliable in every scientific field. The Human+Generated labels are explicitly weak labels for individual pairs, even though Ai2 performed an expert check on 30 selected pairs. The study’s comparison uses particular prompts, models, retrieval configurations, datasets, and stated current API prices, so its exact scores and cost rankings may not transfer to other implementations. It also does not provide a settled definition of novelty or a validated replacement for human scientific judgment.
Skill to build now
Run a sensitivity audit on any LLM judge that gates valuable work. Freeze a representative set of candidates, then vary one factor at a time: prompt wording, model, retrieval context, output format, and aggregation rule. Measure how often the final routing decision changes, not only average accuracy. Escalate unstable cases to human review and retain the input, configuration, evidence retrieved, and decision so that a researcher can inspect why a candidate was accepted or rejected.
Reader takeaway
An LLM novelty verdict is a configuration-dependent judgment, not a ground-truth label. For scientific agents, evaluate the evaluator on the kinds of AI-generated ideas it will actually see, test its sensitivity to reasonable setup changes, and preserve evidence for decisions that affect research priorities.
Open question
What evaluation design can distinguish genuine scientific novelty from plausible recombination while remaining stable across prompts, models, and changing literature?