Run the machine.
Report the exact
final state.
Can a model hold a precise state in its head and update it correctly over many sequential steps, without drifting - no tools, no shortcut? That is the reliability long, multi-step tasks depend on, and the failure mode most benchmarks never isolate.
Leaderboard
Every cell shows two numbers. The large one is per-row accuracy (the default): the share of individual output rows the model got right - a graded score that keeps discriminating even when no model gets a whole grid perfect. The small number below is exact match: the share of grids correct in every row, all-or-nothing. Per-row leads because it is lower-variance and does not saturate; flip the large number with Per-row / Exact below.
Reading a cell: bar = the large metric; †N = N answers cut off by the token budget (counted as wrong); Eff = correct rows per 1,000 tokens the model generated. Click any column heading to sort by it.
What you pay vs what you get
reasoning cost against accuracyAccuracy is only half the picture - it matters how many tokens a model spends to get there. Each row pairs the two: on the left, the median tokens it generates per item (thinking + answer - the cost); on the right, the per-row accuracy it earns (the payoff). A model with a short left bar and a long right bar is efficient - lots of accuracy for little reasoning. A long left bar and a short right bar is wasteful. Whether a model actually finishes within the budget shows up as a † truncation flag in the Overall column above; on the 96k budget almost every model finishes, so it only marks the most verbose reasoners.
What it measures
and why it mattersLong-horizon state tracking.
The model gets the full rules of a deterministic machine, a starting configuration, and a step count, then must carry out every step itself and report the exact final state. The systems are Turing-complete or chaotic, so no formula shortcuts them - the only path to the answer is to actually simulate.
Errors compound. Nothing is judged.
A single wrong cell early on corrupts every later step, so a score reflects sustained, error-free execution rather than a lucky partial. The answer is checked byte-for-byte against a reference simulator - no LLM judge, no rubric. And with no trick to discover, a stronger model can only score higher by tracking more steps correctly, so the benchmark does not saturate.
It predicts real reliability.
Following a multi-step procedure, executing an algorithm, or holding state across a long tool-using trajectory all lean on the same skill. A model that loses the thread at step 15 here will lose it at step 15 of a real workflow. Tools normally hide this - a code interpreter makes every task trivial - so we take tools away to measure the raw capability.
What a score is not. This is not a tool-use, agentic, or general-capability score. Tools are off by design, so a low score means a model simulates poorly by hand - not that it is weak with tools; a strong tool-using model can score low here and still excel in a tool-enabled setting. Scores are genuine, not formatting artifacts: on the lowest-scoring vendor every response finished cleanly and every grid parsed exactly as written, so the misses are real drift, not answers we failed to read.
Tasks
two CA familiesEach prompt contains everything needed: the full rule, the starting state, and how many steps to run. The model doesn't need to know the automaton in advance - it just applies the rule step by step and reports the final state. There are four tasks in two groups: two run on a 2D grid, two on a single 1D row.
Method & integrity
frozen · reproducibleGenerate
Deterministic items from the frozen seed. Identical questions for every model.
Run tools-off
Thinking on, one generous fixed token budget (96k), no code interpreter.
Verify completion
Only genuinely completed responses count. Truncation and timeouts are recorded, never scored as wrong.
Score in code
Byte-exact comparison against the reference simulator. No LLM judge, ever.
Frozen. The configuration is an immutable manifest. Re-hardening is a new version, never an edit, so scores stay comparable across models and over time.
Budget (why 96k). v1 used 64k and truncated verbose reasoners, penalising them for length rather than accuracy. A budget ablation settled it: raising 64k→96k lifted Sonnet 4.6 from 16% to 67% and minimax-m2.5 from 12% to 24% exact, with nothing else changed. 96k captures essentially all reasoning (max output observed ~107k); v2 adopts it, and truncations are still reported for the few models that exceed even that.
Contamination. Answers are generated at runtime, never committed. The canonical set is seed 42. Any other seed mints a structurally identical fresh holdout that has never touched an API, so comparing canonical against fresh accuracy exposes a model that memorised.
† n marks n responses that hit the token budget before finishing. Counted as not correct, but flagged because it is a capacity limit, not a reasoning error.