{"type":"post","id":"turns-out-le-chonk-is-into-rocket-science-3f9b33","slug":"turns-out-le-chonk-is-into-rocket-science-3f9b33","title":"Turns out le Chonk is into rocket science.","description":"Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs.","retrieval_nugget":"Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs.","published_at":"2026-10-16","updated_at":"2026-10-16","record_date":"2026-10-16","date_kind":"published_at","topics":["evals","verification"],"source_urls":["https://www.ulam.ai/stembench"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:turns-out-le-chonk-is-into-rocket-science-3f9b33","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:c207e7bd055727f0ec0a412047b93cb6c78fd384e48028c6c642da92d4583614","reviewed_at":"2026-10-16T12:08:14.601Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"A benchmark table is useful only when its unit, scoring scale, and run conditions remain visible.","observed_facts":[{"text":"Ulam’s STEMBench page describes six domains with 20 tasks each, or 120 tasks per full run, and says the audit aggregates 10 runs.","source_urls":["https://www.ulam.ai/stembench"]},{"text":"Mistral Large 4’s displayed closed-book result is 2.501 out of 4 and ranks sixth among ten shown models; the page reports 118 usable responses, seven level-1 results, 111 level-2 results, and no level-3 or full-target results.","source_urls":["https://www.ulam.ai/stembench"]},{"text":"The top displayed result, GPT-6 Astra xhigh, used Codex with tools, while Mistral Large 4 is listed as closed-book.","source_urls":["https://www.ulam.ai/stembench"]}],"mechanism":"The evaluation aggregates fractional progress scores across task responses and domains, while reporting usability and result levels; execution settings differ between entries, which qualifies direct ranking comparisons.","why_now":"A public audit snapshot dated 14 October 2026 turns a promotional-sounding claim about Mistral’s STEM performance into a checkable multi-domain comparison.","implications":["Readers should assess scientific-model claims by task coverage, scoring rubric, and execution setting, not a headline rank alone.","A broad domain score is benchmark evidence, not proof of autonomous success in real-world research."],"evidence_boundary":"The page reports benchmark performance, not demonstrated success in independent real-world research; the cited excerpt does not establish external replication or the benchmark’s predictive validity.","watch_conditions":["Whether independent researchers reproduce the results or connect STEMBench scores to outcomes on real research tasks."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this record examines the public benchmark result and comparison design rather than treating the private discovery claim as verified evidence."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33/","markdown":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33.md","json":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33.json"},"related_patterns":["ai-engineering-evaluations-are-splitting-into-task-specific"],"next_reads":[{"type":"topic","path":"/topics/evals/","url":"https://newruntime.com/topics/evals/","title":"Evals","reason":"Explore the evals topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/evals.md","json_url":"https://newruntime.com/topics/evals.json"},{"type":"topic","path":"/topics/verification/","url":"https://newruntime.com/topics/verification/","title":"Verification","reason":"Explore the verification topic hub.","media_type":"text/html","markdown_url":"https://newruntime.com/topics/verification.md","json_url":"https://newruntime.com/topics/verification.json"},{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}