Field noteOct 16, 20262 min read1 source

Turns out le Chonk is into rocket science.

In one minute

Ulam’s STEMBench audit reports a six-domain model evaluation with 120 tasks per full run, 20 in each domain, and an aggregate of 10 runs.

Mistral Large 4 scores 2.501 out of 4 in the displayed closed-book comparison, ranking sixth among ten models. These scores describe fractional progress, not a percentage of tasks solved or a probability of correctness. The benchmark reports zero full-target successes across all runs.

A benchmark table is useful only when its unit, scoring scale, and run conditions remain visible. The evaluation aggregates fractional progress scores across task responses and domains, while reporting usability and result levels; execution settings differ between entries, which qualifies direct ranking comparisons. For Mistral, the page records 118 usable responses, seven level-1 results, 111 level-2 results, and no level-3 or full-target results. The top displayed result is GPT-6 Astra xhigh running through Codex with tools, so this is not a like-for-like ranking of model weights alone.

A public audit snapshot dated 14 October 2026 turns a promotional-sounding claim about Mistral’s STEM performance into a checkable multi-domain comparison. Readers should assess scientific-model claims by task coverage, scoring rubric, and execution setting, not a headline rank alone. The page reports benchmark performance, not demonstrated success in independent real-world research; the cited excerpt does not establish external replication or the benchmark’s predictive validity. Whether independent researchers reproduce the results or connect STEMBench scores to outcomes on real research tasks.

This starts a distinct analytical record because no prior New Runtime record was supplied: it examines the benchmark’s public result and comparison design, rather than treating the private claim in the discovery text as verified evidence.

Evidence / sources

  1. 1ulam.ai/stembench
All Field Notes

Next reads

Read next

Open the graph
  1. 01topicEvalsExplore the evals topic hub.
  2. 02topicVerificationExplore the verification topic hub.
  3. 03archiveField NotesOpen the latest editorial analysis.
  4. 04source ledgerSource LedgerInspect the public source evidence graph.
Who read this page?Machine requests, hidden until opened

Loading the privacy-safe route aggregate...

Open the JSON contract