OpenAI’s ARC-AGI-3 write-up is useful because it makes a usually hidden fact explicit: agent benchmarks measure the model and the runtime around the model.
The public story is not just “GPT-5.6 Sol scored better.” OpenAI says the official ARC-AGI-3 harness discarded private reasoning after each action and used rolling truncation as the interaction grew. That meant the model kept seeing some move history, but lost the reasoning that produced earlier moves and eventually lost older actions too.
OpenAI rebuilt the run through the Responses API with two settings that match how its own products run long conversations: retained reasoning and compaction.
What changed
The reported result moved from 13.3% on the ARC-AGI-3 public set with the official harness to 38.3% with retained reasoning plus compaction. OpenAI also says the improved setup used 6x fewer output tokens.
That matters because ARC-AGI-3 is not a one-shot answer test. Agents explore unfamiliar 2D games, infer rules, try actions, and need to preserve what they learned. If the harness erases the agent’s private reasoning and drops old context, the eval is partly measuring amnesia injected by the runner.
The important recommendation is therefore operational: when a team evaluates a long-running agent, it has to version the model, API surface, memory policy, compaction policy, prompt, tools, and scoring harness together.
New Runtime Read
This strengthens the pattern that harness architecture outlives model choice.
A model leaderboard without the runtime settings is incomplete evidence. For agent work, the durable unit is the evaluated workflow: what state persists, what gets compacted, what tools are available, how the score is computed, and which parts are allowed to change.
Teams should treat “same model, different harness” as a first-class eval dimension, not as implementation detail.