StateBench benchmark introduced to measure how AI models track changing facts in long documents
Published
A Hugging Face blog post presents StateBench and reports that frontier AI models handle simple fact updates but lose accuracy on harder, longer state-tracking tasks.
A Hugging Face blog post introduces StateBench, a benchmark testing whether language models keep up with changing facts about entities across long documents.
Its four tiers run from values set once to updates where one variable copies another's current value, like tracking a ball under swapped cups.
Claude Opus 5.5, Gemini 3.8 Flash, GPT-6 Sol and GPT-6 Luna did well on the two easiest tiers but scored 13% or lower on the longest hardest-tier problems.
When 275-million-parameter models were trained directly on the tasks, Gated DeltaNet and hybrid versions stayed nearly perfect on the three easier tiers over tens of thousands of assignments, while transformers slipped on longer inputs and never learned the hardest tier.
Why it matters
According to the authors, a clearer picture of which architectures cope with which kinds of updates could help shape the design and training of new models.
Claims & sources
- c1
A Hugging Face blog post presents StateBench, a benchmark for measuring how well language models follow entity details as they change across long documents.
- c7
StateBench is split into four difficulty tiers, and each tier allows more complex kinds of state updates than the previous one.
- c8
In the easiest tier, called r-trivial, a value is set a single time and never changes afterward.
- c11
In the hardest tier, called general finite, one variable may copy the current value of another, much like keeping track of a ball as cups get swapped.
- c17
Four frontier models were evaluated: GPT-6 Sol, GPT-6 Luna, Claude Opus 5.5 and Gemini 3.8 Flash.
- c18
Those frontier models did well on r-trivial and aperiodic tasks, yet their accuracy was 13% or lower on the longest tasks in the general finite tier.
- c20
The researchers built several 275-million-parameter models from scratch using StateBench, then evaluated them on sequences much longer than any seen in training.
- c21
Out to tens of thousands of assignments, Gated DeltaNet and hybrid models remained nearly perfect on the three easier tiers; transformers lost accuracy on longer inputs and never learned the general finite tier.
- c22
The post says transformers could not learn the hardest tier even when trained on it directly.
- c24
In the authors' view, a better grasp of the update types each architecture can cope with may help shape how future models are built and trained.
Original sources
About this AI summary
AI-generated original factual summary by AI Resum, based on the linked source.
No review, permission, license or endorsement by the source publisher is claimed. Source content and trademarks remain with their respective owners.
Swipe up for next1 / 1