Sergei Makarov

Research notes · ARC Prize 2026 · ARC-AGI-3

A fixed model on one GPU: what moves the score

An open-weights agent on a single GPU · updated 16 October 2026

Interim version. Scores, configurations and code are withheld until the competition's code deadline in November; the full write-up with all measurements will be published here then.

The task

ARC-AGI-3 is a benchmark of interactive puzzle games that an agent has never seen. There are no instructions, no stated goal and no reward signal: the agent sees a small coloured grid, chooses moves, and must work out both the rules and the objective as it plays. A level is scored by how close the agent's move count comes to a human's, so wasted moves are punished hard.

The competition runs on fixed hardware — one GPU, no internet, a time limit per run — which makes it a test of engineering judgement as much as of model ability.

What we did

We ran the project as a measurement programme rather than a sequence of ideas. Every change to the agent was compared against repeated runs of an unchanged baseline, with its success criterion written down before the run. A deterministic local replica of the game engine let us replay recorded games and test "what if" questions without spending any compute.

Over the season we tested a long list of improvements to the agent's instructions and tooling: goal checks, scripted exploration, injected rules and facts, voting, search, and several forms of an internal "world model". We also measured where points can come from at all, before building anything, by recomputing the score of recorded games under counterfactual assumptions.

What we learned

1. Measure the noise first

Identical runs of the same agent differ a lot — single games flip between failure and full success. Until that spread is known, every comparison is a guess. Most claimed gains in this competition are smaller than the spread of one build, ours included.

2. Compute the score budget before choosing a lever

A large share of the levels an agent completes is already scored at the maximum, so making it more efficient on levels it can already solve is capped. Reaching one level deeper in each game is worth more than everything else combined. That told us which ideas could matter before we built them.

3. Adding instructions did not help; giving the agent its memory back did

Text added to the prompt changed almost nothing: the model would repeat an injected rule and still not act on it. What the strongest open solutions had in common was not a better prompt but more of the game's own history kept in front of the model. Under a fixed context budget, every added instruction displaces a piece of the agent's own experience.

4. Audit the whole stack before the top layer

For weeks we tuned the agent while the inference server underneath it was starved — most requests were waiting in a queue, and we misread that as slow generation. The order of work we now recommend for any fixed-hardware AI system: model packaging, serving engine and version, how the memory cache is sized and spent, how time is shared between tasks — and only then the agent's prompt and logic. We did it in reverse.

5. With a fixed model, the score is reasoning efficiency times throughput

Once the serving stack was right, our experiments pointed the same way. Configurations that reduced the total amount of reasoning the server could produce lost. Every change to the agent that we could separate from noise acted through one quantity — how much reasoning the model spends per solved level — and moved it only a little. Nothing we varied made an individual move smarter. Tools that measure the server directly, without playing games, were the cheapest way to make decisions.

6. Read a comparison for what it actually differs in

A stronger model we compared against played no better per move than ours; its whole advantage was speed. We first attributed that to its infrastructure instead of asking what infrastructure we could run ourselves. The answer to that question turned out to be most of the gap.

7. Measure your instrument's noise, not only the agent's

A full competition run is too coarse to see small effects, so we built a paired test: two versions of the agent replay the same set of levels from the same starting states. It looked sensitive — until we ran the same version twice and the two runs differed by as much as the effects we had been reporting. A statistical test that compares levels one by one does not absorb a shift of the whole run. Since then, a single-run verdict is a signal to repeat, not a result.

8. Transfer configurations, not components

One team made a large jump with a different configuration of the whole run. We ported its most visible component into our system, and the score fell. A faithful copy of their entire system ran much faster on the same card, yet scored lower than our own build: their weights were nearly as good as ours, but their agent spent far more reasoning per solved level. The parts of a working configuration pay for each other; lifted out alone, a part is just a cost.

9. What the leaderboard measures

ARC-AGI-3 is meant to measure how efficiently a system acquires skill in games it has never seen. A single score cannot tell apart a model that reasons better in a new game from a model trained in advance to carry priors about games of this kind. Both are within the rules. The best publicly documented fixed-model configurations sit well below the top, and the leading teams whose artefacts we could inspect train their own models. We suggest that future rounds report, next to each score, whether weights were trained and on what.

10. Profile before you guess

When we turned to training, the obvious suspects — the compressed weights, the huge embedding table kept in CPU memory — turned out to be cheap. One small component, written in its library as a straightforward reference loop, took most of the time. Rewriting it, and checking that its output was identical, made training an order of magnitude faster. The same lesson as for the server: measure where the time goes before deciding what to optimise.

How the project was run

Coming in November

The full paper with all measurements: the score budget, the serving experiments, the paired test and its noise, the training results and the code.