---
schema_version: "newruntime-agent-readable-v0.2"
type: "post"
stable_id: "post:openai-arc-agi-settings-harness"
slug: "openai-arc-agi-settings-harness"
title: "OpenAI's ARC-AGI-3 Jump Was a Harness Result"
description: "OpenAI's ARC-AGI-3 write-up shows why agent benchmarks measure the model plus the runtime harness: retained reasoning and compaction changed both score and token use."
retrieval_nugget: "OpenAI's ARC-AGI-3 write-up shows why agent benchmarks measure the model plus the runtime harness: retained reasoning and compaction changed both score and token use. OpenAI's ARC-AGI-3 write-up is useful because it makes a usually hidden fact explicit: agent benchmarks measure the model and the runtime around the model. The public story is not just \"GPT-5.6 Sol scored better.\""
status: "published"
published_at: "2026-07-30"
updated_at: "2026-07-30"
record_date: "2026-07-30"
date_kind: "published_at"
topics: ["evals","agent-harnesses","context-engineering","api-design","models"]
source_urls: ["https://x.com/OpenAI/status/2082616641834422740","https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/"]
visuals: [{"id":"openai-arc-agi-settings-harness-nano-banana","kind":"editorial-diagram","role":"hero","src":"https://newruntime.com/images/posts/openai-arc-agi-settings-harness-nano-banana.webp","alt":"Hand-drawn split diagram where a generic harness drops memory while a Responses API-style harness retains reasoning, compacts context, and produces a score jump.","caption":"OpenAI's ARC-AGI-3 result is a reminder that long-running agent evals measure harness memory and context policy, not only model capability.","credit":"New Runtime synthesis from public source inspection","source_url":"https://openai.com/index/how-two-settings-tripled-our-arc-agi-3-scores/","generated_with":"nano-banana-style-imagegen","width":1600,"height":900,"legend":[{"label":"Generic harness","description":"The benchmark runner discarded private reasoning after each action and used rolling truncation as history grew."},{"label":"Retained reasoning","description":"Passing the previous response ID preserves reasoning across turns and tool calls in the Responses API setup."},{"label":"Compaction","description":"Summarizing long context preserved earlier observations better than dropping the oldest messages."},{"label":"Score jump","description":"OpenAI reports 13.3% on the public set with the official harness and 38.3% with retained reasoning plus compaction."}]}]
telegram_message_id: 2780
telegram_url: "https://t.me/qwgai/2780"
telegram_message_ids: [2779,2780]
telegram_delivery_mode: "media_then_text"
telegram_media_url: "https://t.me/qwgai/2779"
routes: {"html":"https://newruntime.com/posts/openai-arc-agi-settings-harness/","markdown":"https://newruntime.com/posts/openai-arc-agi-settings-harness.md","json":"https://newruntime.com/posts/openai-arc-agi-settings-harness.json"}
source_format: "markdown"
---

# OpenAI's ARC-AGI-3 Jump Was a Harness Result

## Retrieval answer

OpenAI's ARC-AGI-3 write-up shows why agent benchmarks measure the model plus the runtime harness: retained reasoning and compaction changed both score and token use. OpenAI's ARC-AGI-3 write-up is useful because it makes a usually hidden fact explicit: agent benchmarks measure the model and the runtime around the model. The public story is not just "GPT-5.6 Sol scored better."

OpenAI's ARC-AGI-3 write-up is useful because it makes a usually hidden fact explicit: agent benchmarks measure the model and the runtime around the model.

The public story is not just "GPT-5.6 Sol scored better." OpenAI says the official ARC-AGI-3 harness discarded private reasoning after each action and used rolling truncation as the interaction grew. That meant the model kept seeing some move history, but lost the reasoning that produced earlier moves and eventually lost older actions too.

OpenAI rebuilt the run through the Responses API with two settings that match how its own products run long conversations: retained reasoning and compaction.

## What changed

The reported result moved from 13.3% on the ARC-AGI-3 public set with the official harness to 38.3% with retained reasoning plus compaction. OpenAI also says the improved setup used 6x fewer output tokens.

That matters because ARC-AGI-3 is not a one-shot answer test. Agents explore unfamiliar 2D games, infer rules, try actions, and need to preserve what they learned. If the harness erases the agent's private reasoning and drops old context, the eval is partly measuring amnesia injected by the runner.

The important recommendation is therefore operational: when a team evaluates a long-running agent, it has to version the model, API surface, memory policy, compaction policy, prompt, tools, and scoring harness together.

## New Runtime Read

This strengthens the pattern that harness architecture outlives model choice.

A model leaderboard without the runtime settings is incomplete evidence. For agent work, the durable unit is the evaluated workflow: what state persists, what gets compacted, what tools are available, how the score is computed, and which parts are allowed to change.

Teams should treat "same model, different harness" as a first-class eval dimension, not as implementation detail.