---
type: "post"
id: "statebench-testing-how-language-models-track-state-changes-f7c04b"
slug: "statebench-testing-how-language-models-track-state-changes-f7c04b"
title: "StateBench: Testing how language models track state changes over long documents"
description: "Ai2 introduced StateBench, a benchmark that tests how models track changing entity states across long sequences, from simple reassignment to dependencies that require carrying updates forward."
retrieval_nugget: "Ai2 introduced StateBench, a benchmark that tests how models track changing entity states across long sequences, from simple reassignment to dependencies that require carrying updates forward. Its results show that strong performance on easier state questions does not guarantee reliable tracking of harder updates as inputs grow, giving builders a more specific way to evaluate long-context systems."
published_at: "2026-11-03"
updated_at: "2026-11-07"
record_date: "2026-11-03"
date_kind: "published_at"
topics: ["claude","gemini"]
entities: ["Ai2","StateBench","Olmo 3","Olmo Hybrid","Gated DeltaNet","Gemini 3.8 Flash","Claude Opus 5.5","GPT-6 Sol"]
related_patterns: ["company-memory-needs-write-loops"]
novelty_label: "same_event"
prior_refs: ["https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-afffa2/"]
source_url: "https://huggingface.co/blog/allenai/statebench"
source_urls: ["https://huggingface.co/blog/allenai/statebench"]
schema_version: "newruntime-agent-readable-v0.2"
stable_id: "post:statebench-testing-how-language-models-track-state-changes-f7c04b"
status: "published"
visuals: []
editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]}
routes: {"html":"https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-f7c04b/","markdown":"https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-f7c04b.md","json":"https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-f7c04b.json"}
markdown_url: "https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-f7c04b.md"
---

# StateBench: Testing how language models track state changes over long documents

## Retrieval answer

Ai2 introduced StateBench, a benchmark that tests how models track changing entity states across long sequences, from simple reassignment to dependencies that require carrying updates forward. Its results show that strong performance on easier state questions does not guarantee reliable tracking of harder updates as inputs grow, giving builders a more specific way to evaluate long-context systems.

## Summary
Ai2 introduced StateBench, a benchmark that tests how models track changing entity states across long sequences, from simple reassignment to dependencies that require carrying updates forward. Its results show that strong performance on easier state questions does not guarantee reliable tracking of harder updates as inputs grow, giving builders a more specific way to evaluate long-context systems.

## What happened
In a Hugging Face article published November 3, 2026, Ai2 introduced StateBench as a benchmark for long-context state tracking: keeping track of information about entities as it changes across a document or sequence. The authors point to familiar applications such as following variable assignments in code or tracking where characters and objects move in a story. The central distinction is whether a model merely needs the latest value or must correctly apply a chain of changes, where missing an earlier update can invalidate the final answer. The article provides links to a technical report, dataset, and code.

StateBench defines four levels of increasing complexity. “R-trivial” tasks use immutable assignments: the model must recover a value that never changes. “Aperiodic” tasks allow reassignment, so the latest value overrides earlier ones. “Soluble” tasks add self-referential updates, such as incrementing a running score. “General finite” tasks add assignment by reference, where one variable takes another’s current value; repeated swaps between objects are one example. The article relates these distinctions to theoretically motivated classes of state-tracking problems. To make the levels comparable, the authors begin with general finite assignment sequences and derive easier versions, keeping the number of variables, distinct values, and update positions the same while changing the assignment types.

The authors first evaluated pretrained open models on several levels. These included two 7-billion-parameter models, Olmo 3, a transformer, and Olmo Hybrid, which adds Gated DeltaNet layers, as well as smaller models. The models had been pretrained on the same data, and none had seen contexts longer than 64,000 tokens during training; most were tested beyond that length. Almost all did well on the simplest tasks, but accuracy on aperiodic tasks fell as inputs grew, and models generally performed at chance on general finite problems. Format also mattered: some models did much better on RULER-style tasks than on equivalent problems framed as spreadsheet edits.

The article reports results for four frontier models: Gemini 3.8 Flash, Claude Opus 5.5, GPT-6 Sol, and GPT-6 Luna. All did well on r-trivial and aperiodic problems, but performance fell sharply on soluble and general finite tasks as sequences lengthened. Accuracy on the longest general finite tasks dropped to 13% or lower. In a separate experiment, the authors trained 275-million-parameter models from scratch on sequences of up to 256 assignments, then tested them on sequences as long as 64,000 assignments. Gated DeltaNet and hybrid models generalized almost perfectly on the first three levels at the tested lengths; transformers degraded on longer inputs and did not learn general finite tasks. On the hardest level, the more flexible Gated DeltaNet variant stayed near perfect to roughly ten times its longest training sequence.

The authors also describe limitations in their theoretical account. Their new proof establishes that Gated DeltaNet with a restricted memory update can solve all aperiodic tasks and certain periodic tasks under the paper’s fixed-precision arithmetic and rounding assumptions. Its exact capabilities remain open. StateBench assumes a fixed number of variables, while real conversations and documents can introduce new people and objects over time.

## What the archive adds
Our earlier [StateBench Field Note](https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-afffa2/) covers this same benchmark and reports that recurrent and hybrid models generalize farther than transformers on harder state-tracking tasks. This is a restatement of a known story, not a new event; the supplied source adds detail on task levels, pretrained and frontier-model results, and the study’s limits.

## Trend signal
StateBench supports the concern in our [company memory needs write loops pattern](https://newruntime.com/patterns/company-memory-needs-write-loops/): long-context systems do not reliably resolve changing information simply because it is present in a large document set. The benchmark makes that concern more specific by separating “find the latest value” from tasks that require carrying state through a chain of updates. It does not establish how systems perform in production workflows.

## Why now
The authors frame StateBench as a response to the growing variety of recurrent and hybrid architectures. As these designs become more common, a benchmark that separates easy state lookup from more demanding update operations can help researchers compare where different models fail rather than treating long context as a single capability.

## Why it matters
For a team building an agent over code, documents, or conversation history, the failure mode is not always that the model cannot find a relevant sentence. It may find an earlier fact but fail to carry later updates through a chain of references. A context-window size or retrieval score alone will not reveal that distinction.

StateBench gives evaluators a vocabulary for writing more diagnostic tests: does a task need the latest assignment, an accumulated value, or a dependency between entities? That can help teams design evaluation cases that resemble their own state-changing workflows. The reported architecture differences are a reason to test actual candidate models and harnesses, not a reason to switch architectures based on a synthetic benchmark. The study also underscores that input format can change scores, so a team should not assume that success on one benchmark framing transfers directly to its spreadsheet, codebase, or long-running agent task. Earlier coverage: [StateBench: Testing how language models track state changes over long documents](https://newruntime.com/posts/statebench-testing-how-language-models-track-state-changes-afffa2/).

## Limits and uncertainty
The article describes benchmark experiments, not a deployment study, and does not show that the reported performance differences translate directly to production agent quality. Some results concern models trained specifically on StateBench, a more favorable setting than ordinary pretraining. The tested tasks keep a fixed number of variables, unlike real settings where new entities appear. The authors also leave open the full theoretical capabilities of restricted Gated DeltaNet, as well as the effects of numerical precision and combining recurrent layers with attention. The article does not establish that any model reliably tracks state across arbitrary documents or conversations.

## Skill to build now
Build a small state-tracking evaluation from a workflow your team already runs. Write short cases where a fact is unchanged, overwritten, incremented, or transferred by reference, then lengthen each case and vary the presentation format. Score exact final state, not just whether the model cites relevant passages. This mirrors StateBench’s useful distinction between update types while testing whether the behavior matters in your own context; treat it as an evaluation aid, not a substitute for the benchmark’s full methodology.

## Reader takeaway
Long context is not one capability. A model may retrieve the latest statement yet fail when the answer depends on a chain of updates or references. StateBench offers a structured way to distinguish those cases, while its results remain evidence about benchmark tasks—not a guarantee of real-world agent performance.

Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json

## Continue in the graph

Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`.

- topic hub: [Claude](https://newruntime.com/topics/claude.md): Explore the claude topic hub.
- topic hub: [Gemini](https://newruntime.com/topics/gemini.md): Explore the gemini topic hub.
- collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis.
- collection: [Sources](https://newruntime.com/sources/): See the original sources this archive cites.
- whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)