---
type: "post"
id: "turns-out-le-chonk-is-into-rocket-science-3f9b33"
slug: "turns-out-le-chonk-is-into-rocket-science-3f9b33"
title: "Turns out le Chonk is into rocket science."
description: "Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs."
retrieval_nugget: "Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs."
published_at: "2026-10-16"
updated_at: "2026-10-16"
record_date: "2026-10-16"
date_kind: "published_at"
topics: ["evals","verification"]
source_urls: ["https://www.ulam.ai/stembench"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:turns-out-le-chonk-is-into-rocket-science-3f9b33"
status: "published"
visuals: []
editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:c207e7bd055727f0ec0a412047b93cb6c78fd384e48028c6c642da92d4583614","reviewed_at":"2026-10-16T12:08:14.601Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0}
analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"A benchmark table is useful only when its unit, scoring scale, and run conditions remain visible.","observed_facts":[{"text":"Ulam’s STEMBench page describes six domains with 20 tasks each, or 120 tasks per full run, and says the audit aggregates 10 runs.","source_urls":["https://www.ulam.ai/stembench"]},{"text":"Mistral Large 4’s displayed closed-book result is 2.501 out of 4 and ranks sixth among ten shown models; the page reports 118 usable responses, seven level-1 results, 111 level-2 results, and no level-3 or full-target results.","source_urls":["https://www.ulam.ai/stembench"]},{"text":"The top displayed result, GPT-6 Astra xhigh, used Codex with tools, while Mistral Large 4 is listed as closed-book.","source_urls":["https://www.ulam.ai/stembench"]}],"mechanism":"The evaluation aggregates fractional progress scores across task responses and domains, while reporting usability and result levels; execution settings differ between entries, which qualifies direct ranking comparisons.","why_now":"A public audit snapshot dated 14 October 2026 turns a promotional-sounding claim about Mistral’s STEM performance into a checkable multi-domain comparison.","implications":["Readers should assess scientific-model claims by task coverage, scoring rubric, and execution setting, not a headline rank alone.","A broad domain score is benchmark evidence, not proof of autonomous success in real-world research."],"evidence_boundary":"The page reports benchmark performance, not demonstrated success in independent real-world research; the cited excerpt does not establish external replication or the benchmark’s predictive validity.","watch_conditions":["Whether independent researchers reproduce the results or connect STEMBench scores to outcomes on real research tasks."],"related_records":[],"new_branch_reason":"No prior New Runtime record was supplied; this record examines the public benchmark result and comparison design rather than treating the private discovery claim as verified evidence."}
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33/","markdown":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33.md","json":"https://newruntime.com/posts/turns-out-le-chonk-is-into-rocket-science-3f9b33.json"}
related_patterns: ["ai-engineering-evaluations-are-splitting-into-task-specific"]
---

# Turns out le Chonk is into rocket science.

## Retrieval answer

Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs.

Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs. Mistral Large 4 scores 2.501 out of 4 in the displayed closed-book comparison, ranking sixth among ten models. These scores describe fractional progress, not a percentage of tasks solved or a probability of correctness. The benchmark reports zero full-target successes across all runs.

A benchmark table is useful only when its unit, scoring scale, and run conditions remain visible. The evaluation aggregates fractional progress scores across task responses and domains, while reporting usability and result levels; execution settings differ between entries, which qualifies direct ranking comparisons. For Mistral, the page records 118 usable responses, seven level-1 results, 111 level-2 results, and no level-3 or full-target results. The top displayed result is GPT-6 Astra xhigh running through Codex with tools, so this is not a like-for-like ranking of model weights alone.

A public audit snapshot dated 14 October 2026 turns a promotional-sounding claim about Mistral’s STEM performance into a checkable multi-domain comparison. Readers should assess scientific-model claims by task coverage, scoring rubric, and execution setting, not a headline rank alone. The page reports benchmark performance, not demonstrated success in independent real-world research; the cited excerpt does not establish external replication or the benchmark’s predictive validity. Whether independent researchers reproduce the results or connect STEMBench scores to outcomes on real research tasks.

This starts a distinct analytical record because no prior New Runtime record was supplied: it examines the benchmark’s public result and comparison design, rather than treating the private claim in the discovery text as verified evidence.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Evals](https://newruntime.com/topics/evals.md): Explore the evals topic hub.
- topic hub: [Verification](https://newruntime.com/topics/verification.md): Explore the verification topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)