{"type":"post","id":"a-team-of-35-researchers-put-together-50-1f091c","slug":"a-team-of-35-researchers-put-together-50-1f091c","title":"A team of 35 researchers put together 50 unpublished physics research problems for AI to solve.","description":"**Thesis:** TheoPhysEval makes autonomous agent performance on bounded theoretical-physics problems measurable, but its reported accuracy does not by itself establish independent scientific discovery.","retrieval_nugget":"**Thesis:** TheoPhysEval makes autonomous agent performance on bounded theoretical-physics problems measurable, but its reported accuracy does not by itself establish independent scientific discovery.","published_at":"2026-10-15","updated_at":"2026-10-15","record_date":"2026-10-15","date_kind":"published_at","topics":["evals","agents"],"source_urls":["https://arxiv.org/abs/2610.16733"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:a-team-of-35-researchers-put-together-50-1f091c","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:728ef20d9dc0e1e69740f9c3896529edf20ac1f6b1969d20b7cc7e42bd289945","reviewed_at":"2026-10-15T10:05:19.345Z","source_evidence_count":1,"verified_claim_count":2,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":2,"implication_count":2,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"TheoPhysEval makes autonomous agent performance on bounded theoretical-physics problems measurable, but its reported accuracy does not by itself establish independent scientific discovery.","observed_facts":[{"text":"The arXiv preprint describes 50 unpublished research-level theoretical-physics problems with verifiable answers, assembled by 35 researchers.","source_urls":["https://arxiv.org/abs/2610.16733"]},{"text":"It reports that GPT-6-Astra harnessed in Codex achieved 98% final-answer accuracy with an average cost under $3 per problem, using literature search and numerical or symbolic scripting without human hints.","source_urls":["https://arxiv.org/abs/2610.16733"]}],"mechanism":"The benchmark turns a claim about research capability into a score over problems with checkable answers. The authors also report only small average accuracy changes when web search and background papers were withheld, and suggest significant memorization of literature.","why_now":"This result gives teams a concrete performance-and-cost datapoint while renewed claims about AI-assisted research make evaluation design consequential.","implications":["A high score can support the narrower claim that an agent handles this set of calculations under the tested conditions.","It does not show that the system selects important open questions, creates durable new theory, or produces breakthroughs."],"evidence_boundary":"The available evidence is an arXiv preprint's abstract; it does not independently establish robustness beyond the 50-problem suite or the causes of the limited change under ablation.","watch_conditions":["Watch for independent replication on new, held-out problems and detailed analysis distinguishing memorized results from genuinely novel reasoning."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied for this benchmark, so this starts a distinct analytical record without asserting a relationship to another record."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/a-team-of-35-researchers-put-together-50-1f091c/","markdown":"https://newruntime.com/posts/a-team-of-35-researchers-put-together-50-1f091c.md","json":"https://newruntime.com/posts/a-team-of-35-researchers-put-together-50-1f091c.json"},"related_patterns":["scientific-agents-need-artifact-evidence"],"next_reads":[{"type":"topic","path":"/topics/evals/","url":"https://newruntime.com/topics/evals/","title":"Evals","reason":"Explore the evals topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evals.md","json_url":"https://newruntime.com/topics/evals.json"},{"type":"topic","path":"/topics/agents/","url":"https://newruntime.com/topics/agents/","title":"Agents","reason":"Explore the agents topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/agents.md","json_url":"https://newruntime.com/topics/agents.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}