The engineering behind an ARC-AGI-3 solution
How a GPU generates tokens — and why that shapes every solution
0. Why engineering matters here
ARC-AGI-3 is played on fixed hardware: one GPU, a time limit, open-weights models. Under those rules much of a solution is engineering — how many tokens the card can produce, and for how many games at once. This page explains, from the hardware up, how a single card turns model weights into tokens.
Card figures are public reference values for the NVIDIA RTX PRO 6000 Blackwell; the time bars in section 6 are a schematic.
1. What the card is made of
Think of a factory: a warehouse (memory), a road from the warehouse to the workshops (the memory bus), a transfer point (the L2 cache) and workshops with workbenches (the compute units). Only the workshops can compute, and only on what lies on their workbench. Everything in the warehouse has to be delivered first.
| Part | What it is | Size / speed |
|---|---|---|
| GDDR7 memory | Chips on the board around the processor. Model weights, the KV cache (each game's history) and everything else live here. | 96 GB, delivers ~1.8 TB per second |
| Memory bus | The wiring between the memory chips and the processor. Every number passes through it before it can be used. | ~1.8 TB/s — moving 1 GB of weights takes ~0.6 ms |
| L2 cache | Fast on-chip memory shared by all workshops. Small: the model does not fit, so weights stream through it. | ~128 MB |
| Compute units (SMs) | 188 "workshops". Each has tensor cores — machines that multiply tables of numbers very fast — and its own "workbench" (registers, shared memory) of a few hundred kilobytes. | hundreds of trillions of multiplications per second in total |
| KV cache | The part of GDDR7 that holds every game's history: for each token read or generated, its attention "keys and values". Produced by the compute units themselves, never loaded from disk. | ~12 GB (≈1 million tokens in fp8) |
How the KV cache gets into memory. The weights arrive from disk once; the KV cache is born on the card itself. When a request arrives, the units read the whole game history (prefill) and, for every token in every attention layer, compute a pair of vectors — a "key" and a "value". These travel over the bus back to the warehouse, into GDDR7. Every new token appends its own pair. In later steps attention reads the needed pairs from memory back to the cores — that is how the model "remembers" the history without recomputing it. If the next request of the same game starts with the same history, the server reuses it from the cache instead of recomputing it — that is prefix caching.
What the KV cache is, and where transformers come in. Almost all modern language models are transformers, and their central mechanism is attention: to choose the next token, the model "looks" at all previous tokens and decides which of them matter now. For this, in every attention layer each token is turned into three vectors:
- query (Q) — "what I am looking for";
- key (K) — "what I can offer";
- value (V) — "what I pass on if I am chosen".
A new token compares its query with the keys of all past tokens, turns the matches into importance weights and adds up the past tokens' values with those weights. The keys and values of past tokens never change — so there is no need to recompute them every step; it is enough to compute them once and keep them. That store is the KV cache (key–value). Without it, every new token would require running the whole history through the model again, and generation would slow down with every word.
The price is memory: the cache grows with the length of the history, the number of attention layers and the number of concurrent requests. That is why it decides how many games, with how much history, fit on the card — and why it is often compressed (8 or 4 bits instead of 16). The model most teams use in this competition (Qwen3.8-Flash-Next) is a hybrid transformer: only every fourth layer, 12 of 48, keeps a KV cache — these are full-attention layers, and in each step they look at only 2,048 selected tokens of history. The other 36 layers use "linear attention": instead of a growing cache they keep a fixed-size state into which the history is folded, much like a recurrent network.
2. What "reaching the cores" means
A core can only multiply numbers that are already on its workbench. A model weight sits in the warehouse, in memory. So every weight needed in a step makes a trip: memory → bus → L2 cache → the unit's workbench. Only then can a core use it. That trip is what "reading the weights" means.
The key property: a delivered weight can be applied to many requests at once. If the workbench holds the work in progress of 10 games, one weight takes part in all ten computations; with 60 games, in all sixty. Same trip, more useful work.
3. The memory ladder: closer to the cores means faster and smaller
The cores do have their own memory, but it is tiny. Card memory is built in steps: at the top, next to the cores, tiny and instant; at the bottom, on the board, huge and slowest. Bar width below shows capacity on a logarithmic scale (otherwise the top steps would be invisible).
| Step | Capacity | Speed | What it holds during a step |
|---|---|---|---|
| Registers | ~256 KB per unit, ~48 MB per card | one value per clock | the numbers a core is multiplying right now |
| Shared memory / L1 | ~128 KB per unit, ~24 MB per card | very fast | the slice of weights the unit is processing, and the requests' activations |
| L2 cache | ~128 MB | several TB/s | whatever just arrived from GDDR7 — a staging area |
| GDDR7 | 96 GB | ~1.8 TB/s — the slowest link | all model weights (~70 GB) and the KV cache — every game's history |
The main consequence: the model (~70 GB) is hundreds of times larger than all on-chip memory (~200 MB). So in every step the weights pass through the units in transit — a slice is delivered to the workbench, used against every activation, and its place is freed for the next slice. Activations are tiny (kilobytes) and stay at the top. Game histories (the KV cache) sit at the bottom, in GDDR7, and are read every step too — but this model's sparse attention looks at no more than 2,048 selected tokens of history, so a long history barely slows a step down.
Capacities are reference values for the RTX PRO 6000 Blackwell (188 units × 256 KB of registers and × 128 KB of shared memory); the L2 speed is an order of magnitude.
4. Where the model's structure lives
A model is two different things: its structure (how the computation is organised) and its weights (the numbers). The structure is a recipe: short, kept in the computer by the "cook" — the ordinary CPU. The weights are the ingredients: tonnes of them, stored in the card's warehouse. Every step, the cook follows the recipe and tells the card: "bring these weights to the workbenches and do this with them".
| What | Where it lives at run time | Size |
|---|---|---|
| model description (config.json) and model code | on disk; once started, in the computer's RAM, inside the server process | kilobytes–megabytes |
| the "conductor" — the program that drives the card step by step | the computer's CPU (Python, the SGLang scheduler) | — |
| CUDA kernels — small programs for the card (Marlin expert multiplication, attention, etc.) | card memory and the units' instruction caches | megabytes |
| CUDA graph — the recorded command sequence of one step | card memory | megabytes |
| weights | card memory (GDDR7) | ~70 GB |
| KV cache — game histories | card memory | ~12 GB |
| linear-attention states; the draft model; CUDA graphs and work buffers | card memory | ~3 GB; ~4 GB; a few GB |
How it works: at start-up the server reads config.json, builds the model's "skeleton" from the code and fills it with the weights from disk — moving ~70 GB into card memory once, which takes minutes. Then, so that the CPU does not have to issue thousands of small commands every step, the server records the whole command sequence of a step into a CUDA graph once; from then on, every step the CPU says a single word to the card — "replay the graph" — and the card runs all 48 layers by itself. Server logs show this as "CUDA graphs cover 100% of steps".
5. One generation step
- Activations sit on the workbenches. For each request in the batch (for example, 10 games played at once) there is a small vector of numbers — its current state. It is tiny, a few kilobytes, and never leaves the chip.
- Layer 1: deliver the attention and router weights. They are shared by all requests. The cores apply them to every activation and also consult the KV cache — that game's history.
- The router picks experts. Each request needs its own 10 of the layer's 512 experts. Only the chosen ones are delivered — but different requests choose differently, so the bigger the batch, the more distinct experts must be delivered (about 90 of 512 for 10 requests, about 350 for 60).
- The cores compute each activation through its experts. The result stays on the workbench and becomes the next layer's input.
- Repeat for layers 2…48. Each layer's weights travel over the bus again: the whole model does not fit in L2.
- The output layer turns each activation into word probabilities, and the next token is chosen. The step is done: every request has one more token (several, with a draft model).
In total, gigabytes of weights cross the bus in a step, and the result is a single token per request. So the step time is set by whichever takes longer: delivering the weights or computing with them.
6. Why batching and a draft model speed things up — and where the limit is
Pick a mode. The bars show where one step's time goes: blue — cores waiting for weights, green — cores computing. schematic, proportions illustrative
7. What follows for a competition solution
A fixed card can be kept busy in two ways, and they substitute for each other rather than add up.
Few fast streams. Serve the full model to a handful of games at once and fill the idle compute with a draft model. Each game gets fast, high-quality generation; the card is limited by how fast weights can be delivered.
Many slower streams. Compress the model — fewer experts, fewer bits per weight, a lower-precision history cache — so that many more games fit into memory at once. Each delivered weight then serves many requests, the cores do useful work instead of waiting, and the card produces far more tokens in total, even though each game runs on a slightly weaker model and without a draft.
In an interactive benchmark where the score grows with how much an agent can try in the time available, the second road can win — but only as a whole configuration. Its parts pay for each other: memory freed by compressing the weights is spent on concurrency, and the scheduler that shares time between games has to be designed for that concurrency. Lifting one part out on its own, for example pruning experts without the rest, does not carry the gain with it.
Card figures: public reference values for the NVIDIA RTX PRO 6000 Blackwell. The bars in section 6 illustrate proportions; they are not measurements. Measured results from our own runs will be published with the full write-up after the competition's code deadline.