{"type":"post","id":"ai2-has-released-old-ideas-novel-problems-42d2cf","slug":"ai2-has-released-old-ideas-novel-problems-42d2cf","title":"Ai2 has released Old Ideas, Novel Problems","description":"Ai2 has released Old Ideas, Novel Problems, a benchmark for testing whether LLMs can reliably judge the novelty of scientific research ideas.","retrieval_nugget":"Ai2 has released Old Ideas, Novel Problems, a benchmark for testing whether LLMs can reliably judge the novelty of scientific research ideas. Its results show that seemingly modest choices in prompts, retrieval, reasoning budget, and setup can reverse novelty verdicts, making automated research evaluation substantially less dependable than a single score may suggest.","published_at":"2026-10-27","updated_at":"2026-10-27","record_date":"2026-10-27","date_kind":"published_at","topics":["claude"],"entities":["Ai2","Old Ideas, Novel Problems","GPT-5.4","GPT-5.6 Sol","Claude Opus 4.6","Idea Novelty Checker","AI Scientist","ICLR 2026"],"related_patterns":["scientific-agents-need-artifact-evidence","verification-bandwidth-is-the-scarce-resource"],"novelty_label":"new_fact","what_changed":"Ai2's new benchmark tests LLM judgments of research-idea novelty and finds verdicts vary with small changes to the evaluation setup.","prior_refs":["memory:96975f75-4b66-8eef-aa9c-6e62813ca3dc"],"source_url":"https://allenai.org/blog/novelty-bench","source_urls":["https://allenai.org/blog/novelty-bench"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:ai2-has-released-old-ideas-novel-problems-42d2cf","status":"published","visuals":[],"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/ai2-has-released-old-ideas-novel-problems-42d2cf/","markdown":"https://newruntime.com/posts/ai2-has-released-old-ideas-novel-problems-42d2cf.md","json":"https://newruntime.com/posts/ai2-has-released-old-ideas-novel-problems-42d2cf.json"},"next_reads":[{"type":"topic","path":"/topics/claude/","url":"https://newruntime.com/topics/claude/","title":"Claude","reason":"Explore the claude topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/claude.md","json_url":"https://newruntime.com/topics/claude.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}