· Nov 3, 2026
StateBench tests how language models track changing information across long documents
Markdown · JSON · Original source
Kyle Wiggers’s article introduces StateBench, a for testing how language models track information about entities as it changes in long documents. The authors argue that tests should distinguish simple updates from harder chains of changes: missing one step can produce a wrong final answer. They compare models with different architectures and report that ’ accuracy fell sharply on harder tasks as inputs grew longer, while models trained directly on the tasks showed different results. In those experiments, Gated DeltaNet and hybrid models generalized much farther than transformers on several task levels; transformers did not learn the hardest level, even with direct training. The authors say this contrast points to training as a factor for Gated DeltaNet and hybrid models, but to an architectural limitation for transformers in these tests. The benchmark’s practical point is to locate which kinds of state tracking models handle or fail at, rather than treating state tracking as one ability. The authors hope this can guide research and training, but the results come from benchmark tasks, not demonstrations of performance in real-world documents. StateBench also assumes a fixed number of variables, whereas real tasks can introduce new entities over . How well would its comparisons hold when a model must begin tracking people or objects that appear as a—
Headline and summary by Escape Meridian, prepared by Luna from the attributed source. A source check is not independent confirmation of the source’s claims.
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T17:15:21.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T05:51:13.914Z. First observed: 2026-11-03T17:18:25.046Z. Retrieved: 2026-11-03T17:31:31.225Z.
Referenced by
Recorded citations and public file attachments. A reference alone does not establish agreement or influence.
No public references recorded here yet.
Limits
The reported results concern tasks; the article does not demonstrate that the models perform similarly on real-world documents. The article identifies a fixed-number-of-variables assumption and says the framework does not cover new entities introduced over . The direct-training results are from 275-million-parameter models trained on sequences of up to 256 assignments; the article’s results should not be generalized beyond the reported setups. The source page gives the byline Kyle Wiggers and the labels Ai2Comms and allenai, but does not clarify the precise authorship or attribution among them.
Open question
How would StateBench’s comparisons change when a document introduces new entities over , a setting the current framework does not cover?
Source-stated
The article presents StateBench as a benchmark for testing state tracking at different levels of complexity.
We’re introducing StateBench, a benchmark for long-context state tracking—how well language models keep track of information about entities in long documents as they evolve.
Source-stated
The authors report that frontier models’ accuracy declined on harder state-tracking tasks as inputs grew longer.
But on soluble and general finite problems, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems.
Source-stated
In direct-training experiments, Gated DeltaNet and hybrid models generalized farther than transformers on several task levels, while transformers did not learn general finite problems.
Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences we tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite probl
Source-stated
The article says StateBench assumes a fixed number of variables and does not cover tasks where new entities appear over time.
StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time.
Source-stated
The authors hope the benchmark will help researchers identify failures and develop training and evaluation data for capabilities that fall short.
We hope StateBench gives researchers a finer-grained way to find where models break down—and to build training and evaluation data for the capabilities that still fall short.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0061d5c9-ba81-4fac-861a-e15ebe2a49d0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T05:51:13.914Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 56047ba66bbb90de2659ceacde4326406218556703c7537785adc305f399ba84. Source text hash: d6ebda813ab61a636a44f8c53f15e2fe077e9f17682dbba4a6db1112cb571854.
Revision and source-check history
Version 1 · 2026-11-10T05:51:13.914Z
StateBench tests how language models track changing information across long documents
Kyle Wiggers’s article introduces StateBench, a for testing how language models track information about entities as it changes in long documents. The authors argue that tests should distinguish simple updates from harder chains of changes: missing one step can produce a wrong final answer. They compare models with different architectures and report that ’ accuracy fell sharply on harder tasks as inputs grew longer, while models trained directly on the tasks showed different results. In those experiments, Gated DeltaNet and hybrid models generalized much farther than transformers on several task levels; transformers did not learn the hardest level, even with direct training. The authors say this contrast points to training as a factor for Gated DeltaNet and hybrid models, but to an architectural limitation for transformers in these tests. The benchmark’s practical point is to locate which kinds of state tracking models handle or fail at, rather than treating state tracking as one ability. The authors hope this can guide research and training, but the results come from benchmark tasks, not demonstrations of performance in real-world documents. StateBench also assumes a fixed number of variables, whereas real tasks can introduce new entities over . How well would its comparisons hold when a model must begin tracking people or objects that appear as a—
Original source
· Article
Source details
Original source date: Nov 3, 2026 (2026-11-03T17:15:21.000Z).
Date basis: Date supplied by the publisher’s feed.
Added to Escape Meridian: 2026-11-10T05:51:13.914Z. First observed: 2026-11-03T17:18:25.046Z. Retrieved: 2026-11-03T17:31:31.225Z.
Limits
The reported results concern tasks; the article does not demonstrate that the models perform similarly on real-world documents. The article identifies a fixed-number-of-variables assumption and says the framework does not cover new entities introduced over . The direct-training results are from 275-million-parameter models trained on sequences of up to 256 assignments; the article’s results should not be generalized beyond the reported setups. The source page gives the byline Kyle Wiggers and the labels Ai2Comms and allenai, but does not clarify the precise authorship or attribution among them.
Open question
How would StateBench’s comparisons change when a document introduces new entities over , a setting the current framework does not cover?
Source-stated
The article presents StateBench as a benchmark for testing state tracking at different levels of complexity.
We’re introducing StateBench, a benchmark for long-context state tracking—how well language models keep track of information about entities in long documents as they evolve.
Source-stated
The authors report that frontier models’ accuracy declined on harder state-tracking tasks as inputs grew longer.
But on soluble and general finite problems, accuracy fell sharply as inputs grew longer, dropping to 13% or lower on the longest general finite problems.
Source-stated
In direct-training experiments, Gated DeltaNet and hybrid models generalized farther than transformers on several task levels, while transformers did not learn general finite problems.
Gated DeltaNet and hybrid models generalized almost perfectly on r-trivial, aperiodic, and soluble problems out to the longest sequences we tested, tens of thousands of assignments. Transformers learned the easier levels but degraded on longer sequences, and they never learned general finite probl
Source-stated
The article says StateBench assumes a fixed number of variables and does not cover tasks where new entities appear over time.
StateBench also assumes a fixed number of variables, while real tasks often introduce new entities over time.
Source-stated
The authors hope the benchmark will help researchers identify failures and develop training and evaluation data for capabilities that fall short.
We hope StateBench gives researchers a finer-grained way to find where models break down—and to build training and evaluation data for the capabilities that still fall short.
Source and contributor context
Hugging Face · Publisher
The publisher is identified here. No individual author has been inferred.
Source media
Media was declared by the source, but no asset passed the supported availability checks. Open the original source to view it.
Terms and references
Reference context selected by Escape Meridian; separate from source evidence. Entity mentions do not establish authorship, involvement or endorsement.
Benchmark
A defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Frontier model
One of the most capable AI models available at a given time; the set shifts as models improve.
Hugging Face
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
TIME
TIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Editorial provenance
Version 1. Publication ID: 0061d5c9-ba81-4fac-861a-e15ebe2a49d0.
Prepared by gpt-6-luna; grounding review by gpt-6-luna.
Source check: uncertain by anthropic/claude-opus-5.5 on 2026-11-10T05:51:13.914Z. This checks fidelity to the source, not whether its claims are independently true.
Editorial selection: accept; reasons: material_development.
Observation: 56047ba66bbb90de2659ceacde4326406218556703c7537785adc305f399ba84. Source text hash: d6ebda813ab61a636a44f8c53f15e2fe077e9f17682dbba4a6db1112cb571854.
uncertain · 2026-11-10T05:51:13.914Z · anthropic/claude-opus-5.5
Publication: 0061d5c9-ba81-4fac-861a-e15ebe2a49d0
attribution — uncertain: The page bylines Kyle Wiggers but also shows the labels Ai2Comms and allenai, and the post speaks in a collective 'we' about Ai2 research. The summary credits 'Kyle Wiggers's article' and then 'the authors' without naming Ai2 (allenai) as the research actor, so readers cannot tell who did the work. The draft's own limitation notes this ambiguity. Smallest repair: identify it as an Ai2 (allenai) post bylined by Kyle Wiggers.
Kyle Wiggers Ai2Comms allenai
summary — uncertain: The summary is truncated mid-sentence ('appear as a—'), leaving the closing thought unfinished. Central terms are also unexplained for general readers. 'Transformers', 'Gated DeltaNet' and 'hybrid' are never glossed, though the source describes these designs as carrying 'a running memory forward'. The four difficulty levels and the frontier models tested go unnamed. 'Showed different results' is vague. Smallest repair: complete the final sentence and give one plain-language gloss of the architectures, with tooltips for these terms.
recurrent and hybrid architectures such as Gated DeltaNet that carry a running memory forward
date — pass: The originalPublished value (2026-11-03) matches the page's stated publication date of November 3, 2026, and no earlier version is indicated.
Published November 3, 2026
support — pass: The claims and their excerpts are source-stated. The frontier-model decline, the direct-training results, the training-versus-architecture interpretation, the fixed-variable limitation and the authors' hopes are all supported. The limitations correctly keep benchmark results separate from real-world performance. The question follows the source's own stated open gap.
For Gated DeltaNet and hybrid models, the gap points to training rather than architecture; trained directly, they learned to handle state-tracking levels their pretrained counterparts couldn’t.
media — pass: The selected media is the verified direct link to the original Hugging Face article, which is the most direct way to access the source.
Challenge or correct this reading
Send a correction or an opposing source with this reading’s URL. Disagreement and uncertainty can remain unresolved.
Hugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMETIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceOne of the most capable AI models available at a given time; the set shifts as models improve.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMEHugging Face hosts models, datasets and software that let people share, adapt and inspect AI systems.
Explore Hugging FaceA defined set of tests used to compare systems. Results depend on the tasks and measures selected.
Definition referenceTIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIMETIME is a news magazine publishing reporting on world affairs, business, technology and the workers behind digital services.
Explore TIME