What Moved the Score on ARC-AGI-3: A Measured Account
Revision history
Draft status, 01.10.2026: revised after the Milestone 2 disclosures (§12). The earlier title, "Scaffolding Multiplies Ability It Cannot Create", and the conclusion of §8 located the ceiling in the model; the disclosed solutions refute that reading, and our unmodified copy of the best open solution scored 24.25 against 4.30 for our own best build.
Update, 06.11.2026: synchronised with the Paper Track submission (submission 57859496; battle composition, the pulse count over 13 battles, the one-battle threshold over all 17 family battles: ±6).
Update, 30.10.2026: §1A added — where the score came from, source by source (base, model, harness, server, platform, measurement), with Figures f15–f19, checked against the raw sources by an independent reader; §13G result (29.10) and §13H at fifteen battles (30.10); stale and overstated statements across the draft and the writeup corrected (runs/paper_sources_3010/draft_corrections.md, check_section.md).
Update, 28.10.2026: §13G extended (RL shortcut removed by action-number permutation; result pending) and §13H added — fourteen battles (29.10) of one configuration and an audit of the harness itself. The 1,500-word writeup was revised accordingly.
Update, 16.10.2026: §13D added — the level-start twin test and its measured noise, battle repeats, training speed on one card (vectorised sparse-attention indexer, ×13). The 1,500-word writeup (docs/writeup_1500.md) was rewritten around the fixed-model thesis.
Update, 14.10.2026: §13C added — what the leaderboard measures: a fixed model against a model trained for games.
Update, 10.10.2026: §13B added — what changed our picture between 06.10 and 10.10: a transplanted component fails where the whole configuration works; two ways to fill the card (concurrency against speculative decoding); a fine-tuned draft model whose small gain was predictable from logs we already had; live test-time training measured as harmful; a larger KV cache that is stable but buys nothing at ten games; and what is and is not known about the top of the leaderboard. Figures for this section are kept in the project log and will be added for the final version.
Update, 03.10.2026: §13 added — serving experiments on the disclosed stack (a replay bench for the server, a KV-quality probe, where the cache starts to bind, and our NVFP4 KV-cache patch for the model's sparse attention); §11 item 9 (what we misread in the teacher comparison); §12 updated with a second submission of the copy and two 25-game controls.
Code: battle notebook arc3-graft-dfranzen-m2-cbattle, version 3 (the 30.97 submission; later versions add the goal note and server watchdog of §1A.3); early kernels arc3-stock-flash (baseline) and arc3-stock-flash-predhint (goal-predicate layer) are not public.
Abstract
We ran a five-week ablation programme on ARC-AGI-3 with a fixed open-weights agent (Duck harness + Qwen3.8-Flash-Next-NVFP4 on one 96 GB card), served throughout with a 32,768-token history window. Twenty harness layers produced no score change distinguishable from run-to-run spread. We localised the failure — the model induces correct mechanics but not goals — and formalised the level goal as a machine-checkable predicate. Under an honest control (a random non-goal state declared the goal, parameters re-fitted by the same search) the vocabulary separates true completion states on 83% of levels against 51% for pseudo-goals. Leave-one-game-out prediction of the next level's goal succeeds in 27–35% of 26 level pairs against 0–2% for two nulls. The decisive measurement is a goal-conditioned progress function: it separates winning from sibling actions at AUC 0.63 given the true completion frame and at chance (0.47) given the goal transferred from the previous level — the only goal available in play.
Two further contributions are aimed at anyone choosing where to spend effort on this benchmark. First, a score budget: 64% of the levels our agent completes are already at the scorer's cap, so all efficiency work is bounded at +24%, clearing a first level in every game is worth +32%, and one additional level per game is worth +137% — the only lever that exceeds the current score, and the one none of our harness layers moved. Two ideas were tested in this arithmetic: two short passes with best-of-runs scoring lose 19% against one long pass, and giving a game more time under a fixed per-game cap is bounded near +4% (a bound on that intervention in our serving regime, not on priority scheduling between games, which we did not test). Second, changing the core model is measured rather than assumed: a stronger model crushed to 2–3 bits to fit the card is worse than the baseline (1 level against 3–5), an uncompressed stronger model through the same harness is far better (10 levels in ~14–15 minutes, cut short by billing, against 4/5/3 in 30 minutes), and we document, with the exact constraints, two working ways to deliver modified weights into the sealed competition kernel — a runtime adapter that costs four serving concessions, and a byte-level weight overlay that costs nothing at all.
The most important result is a correction of our own conclusion. We read twenty null layers and two model swaps as evidence that the model was the ceiling. Milestone 2 showed otherwise: four disclosed solutions reached 20.00–27.89 on the public leaderboard against our 4.30 with the same model family, the same card and the same harness lineage. What they share is not a prompt or a layer but memory — a history window of 70–131 thousand tokens, made affordable by a compressed KV cache, prefix caching and trimming in large blocks — and one of them moved from 14.49 to 22.53 by that change alone, with no prompt edited. Our layers were measured in a regime where the agent kept a quarter to a half of that history and the server computed about three requests at a time; we had measured that shortening history hurts (§2) and never tested lengthening it. The lesson we can state with evidence is about order of work: in a fixed-hardware competition, how much of its own history the agent keeps is decided by the serving stack, and that has to be settled before any harness layer is worth measuring.
1. Setup
All measurements use the 25 public games, the Duck harness (Milestone 1 winner) and Qwen3.8-Flash-Next-NVFP4 on one RTX Pro 6000 — the configuration the public leaderboard converged on. A deterministic local replica reproduces recorded trajectories exactly (561/561 actions; per-level counts matched official counters on 121/121 levels), making counterfactual analysis free.
Baseline spread is the unit of measurement. Three runs of the identical configuration score 1.67 / 2.73 / 3.32 at a 30-minute per-game cap and 5.46 / 6.32 / 7.23 at 80 minutes; builds on this base scored 1.85–4.09 across ten competition submissions, and the unchanged base itself 3.36 / 2.31 / 3.23 (06–09.09; mean 2.97, SD 0.57). Any claim ignoring this spread is unfalsifiable, so every probe was pre-registered against three truncations of base runs.
The serving regime is a condition of every result in §§2–8. All of them were taken with the profile inherited from the public line: a 32,768-token history window, a 5 GiB BF16 KV cache (about 105 thousand tokens), prefix caching off, history trimmed exactly to fit on every request, and — as we found only at the end of the month (§11) — about three requests computed at a time out of sixteen slots. The disclosed Milestone 2 solutions run the same model with a KV pool of 1.0–1.4 million tokens and a 70–131 thousand token window (§12). A null result below is a null result in the first regime; whether it survives in the second is untested unless stated.
1A. Where the score came from: base, model, harness, server, platform, measurement
Added 30.10.2026, built from runs/paper_sources_3010/{model,harness,server,platform_measure_base}.md and checked against the raw sources by an independent reader the same evening (runs/paper_sources_3010/check_section.md). Items marked [31.10: pending] await runs not finished at the time of writing.
Our best battle score is 30.97 (01.11), 654th of 5,029 teams on the leaderboard downloaded that day; the tenth place stood at 46.82 and the first at 75.56 [measured]. The previous best, 30.85 (28.10), was 551st of 4,930 [measured]. This section says where those points came from, source by source, before the chronological sections. The answer has three parts. First, every large step in our battle score came from changing the base we built on, and every base was built by others: 1.61 → 3.36 when a public kernel swapped the model under an unchanged Duck harness (06.09), 4.30 → 24.25 when we submitted an unmodified copy of the best disclosed Milestone 2 solution (01.10) [measured; Fig. f15]. Second, everything we added to that copy sits inside the spread of one build family: fifteen battles of four builds scored 26.08 ± 2.85 and the build means (25.1–27.2) do not differ (one-way ANOVA p = 0.71); a fifth build scored 24.01 in four battles (Welch p = 0.28 against the fifteen) [measured; Fig. f04]. Our best score is the upper edge of that spread, not a contribution [inferred]. Third, within one base the only quantity we saw move the score is total generated tokens, which the serving stack decides: halving generation roughly halved the bench score at unchanged value per move, and in September a starved KV cache held our server to about three requests at a time [measured]. We saw this lever work only downward; none of our own server additions (+1.5 to +2.4% tokens/s at equal load) is visible in battle, and at one battle's resolution none could be [inferred]. The platform cost points we cannot count — weight loading of 4 to 74 minutes, and three zero-score runs whose cause, ours or the platform's, no log can establish — and measurement added none, but decided which of our claims survived.
| Source | What we did | Distinguishable contribution to the battle score | Where |
|---|---|---|---|
| Base (whose foundation) | Adopted in turn Duck (Tufa Labs), keithtyser's Flash-Next kernel, Scott Le Grand's harness fixes, D. Franzen's disclosed Milestone 2 configuration | Two large steps — 1.61 → 3.36 and 4.30 → 24.25 — both base changes made by others [measured] | 1A.1; Fig. f15, f01, f02 |
| Model and its training | Compared models and packagings; fine-tuned the draft model; ported expert pruning; copied a pruned 3-bit build; trained LoRA adapters; RL in miniature | None positive of ours. Draft inside the spread; pruning 22.65; 3-bit build 19.52; trained adapter 15.73, harmful [measured] | 1A.2; §8–10, §13B, §13E–G; Fig. f09, f18 |
| Harness | ~20 layers in September; brevity, macro moves, notes, goal note, keep-alive pulse, watchdog in October; an audit | None distinguishable. Removing past reasoning from history: 9.43 → 2.91 [measured]; the pulse is insurance, not score | 1A.3; §2, §12, §13A, §13D, §13F, §13H; Fig. f19, f07, f12 |
| Server | Audit of the September stack; replay bench, KV probe, NVFP4-KV patch, draft fine-tune, pruning port, profiles, SGLang 0.5.22 port | The disclosed stack came with the copy (not separated from its harness). Ours: +1.5 to +2.4% tokens/s at equal load, none visible in battle; one 0.00 [measured] | 1A.4; §11, §13, §13A–B; Fig. f17 |
| Kaggle platform | Diagnosed loading, the idle-time rule, zero runs, start hour; added a pulse and a longer server wait | Negative, size unknown: 3 zeros in 30 October battles (cause not determined); 30 → 75 min of loading ≈ −3 points [assumed] | 1A.5; §13H; Fig. f08, f14, f16 |
| Measurement | Noise floors, decision rules written before runs, a paired level-start test, a score budget, equal-token comparison | No points. Set what each instrument sees (one battle ±6, one long stand ≳12) and withdrew several of our claims | 1A.6; §1, §7, §12, §13D, §13H; Fig. f03–f06 |
1A.1 Base: whose foundation
What we did. Our six submissions before the Duck-based build (16–22.08) scored 0.00–0.07 [measured]. On 23.08 we moved to Duck (Tufa Labs, Milestone 1 winner: one language model with a single Python tool), adding about 70 lines of configuration: 1.19 [measured]. Stock Duck on Qwen3.8-27B-FP8 scored 1.61 (01.09). On 06.09 a public kernel by keithtyser — Duck unchanged, the model swapped for Qwen3.8-Flash-Next-NVFP4 — scored 6.76 on the 25 public games for its author and 9.00 in Tufa's verbatim copy, against 3.02 for our stock run; we submitted it as is and scored 3.36 [measured]. About twenty of our layers on that base then gave battles of 1.85–4.09, none outside run-to-run spread (§2). Scott Le Grand's harness fixes scored 4.11 as published (27.09); with our function memory and sixteen server slots, 4.30 (28.09), our best before Milestone 2 — 0.19 above Le Grand's, inside the spread of the Flash-Next base (SD 0.57, three battles) [measured]. At the close of Milestone 2 (30.09) the disclosed solutions stood at 20.00–27.89 and the top at 50.65 (§12).
On 01.10 an unmodified copy of Franzen's notebook scored 24.25 — ×5.6 in one submission, 63rd of 3,539 — and 26.78 on 02.10 [measured]. Nothing of ours was in it. The copy is itself layered: its server is the Pennyroyal v2.5.3 SGLang fork (jpezzulli) with patches by mratsim, Gabriel Olympie and Franzen; its harness is the Duck / ARC3-Inference line (Jeroen Cottaar, Tufa) with Franzen's patch; its model is Intel's W4A16 AutoRound packaging with the albucino draft [measured, by reading the code]. On 30.10 we compared server launch cells: in 13 of 16 strong public notebooks that run a server the cell is byte-identical to Franzen's; ours differs only by the block that loads our fine-tuned draft [measured]. The notebooks differ in their harness settings.
Transplanted parts did no better than the whole copy. A public fork's re-fitted scheduler scored 27.40, inside the copy cloud (06.10); face-of-agi's expert pruning with 20 concurrent games, 22.65 (07.10); an unmodified copy of face-of-agi's whole p37 build, 19.52 (11.10), against their leaderboard 37.94 [measured]. Our additions on the copy — a fine-tuned draft (mean 25.27, two battles), brevity (27.23, four), the pulse (25.60, seven), a goal field with watchdog (24.01, four) — do not separate from the plain copy (26.61, four); of single-submission builds only a copied whole build (19.52), a trained adapter (15.73) and three zero runs fall outside the band [measured; Fig. f04].
What we submitted. The battle build is Franzen's Milestone 2 notebook — the Duck / ARC3-Inference harness with his patch, the Pennyroyal SGLang fork, Intel's W4A16 AutoRound packaging, ten games at a time, a 128k-token history — plus three changes of ours: the albucino draft fine-tuned by us as a three-step chain (§1A.2), the brevity instruction of §1A.3 and a keep-alive pulse that resets a game every 8 minutes until the server answers, so the 15-minute idle rule cannot end the run while the model loads (§13H) [measured, by reading the notebooks]. Seven battles of this build scored 24.09 / 26.67 / 30.85 / 22.12 / 21.58 / 22.92 / 30.97 (mean 25.60, SD 3.98); our best, 30.97 (01.11, submission 57859496), is the upper edge of that spread [measured]. The final battle (02.11) adds our weight-only FP8 for the large dense projections and the output head (§1A.4): one FP32 scale per output row, the BF16 copies freed so the KV pool grows from 1.01M to 1.26M tokens, and a Triton kernel with a per-shape tiling table, 1.6–1.7 ms faster per verify step at ten games in a microbenchmark [measured]; its minimal pre-battle run converted 36 + 60 projections and the head, raised no errors and cleared 6 levels on three 5-minute games against 5 for the same build without it [measured; too short to compare speed or quality]. From how the score grows with generated tokens we expect under a point on average from the extra speed [inferred]; a single battle resolves only about ±6 (§12), so a result above 30.97 — higher than all 17 earlier battles of the family, a 1-in-18 chance with no effect — would be suggestive, not proof. It scored 20.13 [measured]: below all seven battles of the build without it and the lowest of the 18 in the family — one-sided p = 0.12 against the seven, 0.035 against all 17 (rank 1 of 18, p = 0.056) [measured]. The cause is not established: noise (other builds of the line scored 19.73 and 19.52), the FP8 rounding error (2.65% by norm in the microbenchmark, above our own 2% bar; error building up in the memory layers' gates over long histories is a hypothesis we did not measure), or slower loading (server ready in 832 s against 643 s in the pre-battle runs, on different machines; about −0.2 points by the estimate below) [inferred]. One battle does not tell them apart. After the deadline two other teams reported on the competition forum that FP8 weights in SGLang cost measurable token efficiency, one of them more for output projections than for input projections [forum, not verified]; our conversion covered both and the output head. Code and notebooks: https://www.kaggle.com/code/sergueimakarov/arc3-graft-dfranzen-m2-cbattle?scriptVersionId=360443322 (version 3, the 30.97 submission; later versions add the goal note and server watchdog of §1A.3); final FP8 battle: https://www.kaggle.com/code/sergueimakarov/arc3-graft-dfranzen-m2-cbattlefp8; notebooks under Apache-2.0, the code dataset arc3-mtp-chain-code under CC0.
Rank fell while the score rose: 63rd of 3,539 at 24.25 (01.10), 509th of 4,333 at 30.04 (15.10), 551st of 4,930 at 30.85 (28.10), 654th of 5,029 at 30.97 (01.11); the tenth place moved from about 30 to 46.82, and on 28.10 one point in the 30–32 zone was worth about 250 places [measured; Fig. f02].
What holds. Each large step is a base change; the new bases were repeated (three and four battles), and the steps are about ×2 and ×5.6 against a battle spread of 0.57–0.60 on the earlier bases [measured; robustness inferred]. Within the October base no addition of ours is distinguishable in fifteen battles; no part transplanted from another build beat the plain copy (27.40, 22.65), and neither did a whole copy of a second team's build (19.52) [measured, one battle each]. Not resolved: how the 4.30 → 24.25 step divides between Franzen's server, packaging and harness patches; why the p37 copy scored 19.52 (which of 45 submissions gave its authors 37.94 is unknown); whether harness work alone reaches the top ten.
What we missed. After the deadline the team placed eighth on 01.11 (50.48; sixth on the final public board, 54.55) described its configuration on the competition forum: a patched vLLM server, the model pruned to 400 of 512 experts, a KV cache in int4 for keys and values held in host memory, games never evicted (so no history is prefilled twice), and therefore all 110 hidden games playing at once at about 3,500 generated tokens/s by its write-up (3,300–3,400 including prefill in an earlier post), against our ~600 at ten games; after 5–6 hours, when few games are still live, the full 512 experts are restored, which they credit with four points; the harness is D. Franzen's, almost unchanged [forum, not verified]. We had every part, each measured alone in a setting where it could not pay: expert pruning gave +1.7–2.4% at equal concurrency and +10.6% at 20 streams on a rented GPU, and its only battle (07.10, together with 20 games and no host cache — two changes at once) scored 22.65; our 4-bit KV cache enlarged the pool ×1.5 without likelihood loss but was measured at ten games, where the pool was rarely full; a host-memory cache turned −32% into +7% at 20 games, and 12 games (+1.5%) was not submitted [measured]. The combination — less memory per game in exchange for many more games — was never built. The main decision that steered us away was to keep the copy's SGLang fork as given and stop exploring vLLM. Until 30.09 we had served on vLLM; on 29.09 vLLM 0.30 had added an FP8 KV cache for the model's sparse attention (×1.77 capacity), CPU offload of the n-gram table and native support for a 448-expert pruned checkpoint, and our 25-game bench on it ran 12–15 of 16 requests at once and scored 4.71 against 2.85 for the unpruned model [measured]. From 01.10 none of this was carried over, and on 29.10 we judged newly listed vLLM speed-ups inapplicable because the battle ran on SGLang, instead of running the copy's harness on vLLM, which is what the eighth-placed team did. An early working assumption that tokens per game equal time × throughput / games, which ignores that decode throughput grows with the batch until memory binds, pointed the same way [inferred from our records]. By our own rule, each of these verdicts should have read "not carried through".
Checked after the deadline (04.11.2026). We ran the battle stack's model (Qwen3.8-Flash-Next, Intel W4A16 AutoRound) on stock vLLM (main 40941f2c) on one rented RTX Pro 6000 for about five hours, replaying prompts recorded in a 90-minute run of the battle build on the 25 public games (Kaggle, 31.10) [measured; raw logs and replay outputs kept in our records]. In that run's logs 96.7% of consecutive prompt pairs of a game only append to the history (median shared prefix 95.9%), so the prompts are cacheable [measured, 1,978 consecutive pairs]. Yet at 20 games and at 110 streams (22 games, five copies each) the server's prefix-cache hit rate was 3.0–4.9%, against 92% for SGLang in the recorded run, where 25 games shared ten server slots (pool above 0.9 full in 3% of decode logs); with a single game (four requests) it was 70.1%, with no preemptions [measured]. The cache works; the collapse is games evicting each other's histories from a card that holds about eight of them [inferred]. Launch flags alone, with no code change, grew the cache from 592,830 to 774,256 tokens (GPU memory share 0.975, at most 16 sequences, CUDA graphs up to 64 tokens), and to 1,446,415 tokens with the MTP draft also switched off (32 sequences) [measured; vLLM counts tokens over all cache groups, not comparable with SGLang's 1.01M]. With more of the games fitting, throughput rose: six games ran at 437.8 tokens/s against 218.7 on the smaller cache (hits 83.6% against 43.5%, preemptions 0 against 11), eight games at 228.5 against 171.9 (+33%); ten games on the larger cache ran at 214.3 [measured, one window per point, two server loads; the load-to-load shift was not measured]; twelve games without the draft reached 485.9 tokens/s [measured; Fig. f21]. These are single windows of a few minutes on one machine, not a controlled comparison with our SGLang build, which ran ~600–800 tokens/s at ten games (§1A.4). The public first place, posted the same day, is also built around memory per game: Qwen3.8-Flash-Next pruned 512 → 320 experts, 3-bit expert weights with quantization- and pruning-aware distillation from the full model, an n-gram table in 5 bits in host memory, a private SGLang fork with sparse-attention KV history in host memory, an NVFP4 KV cache and its own kernels — an effective 11.7M-token pool, 110 concurrent games, ~3,800 tokens/s end to end; reinforcement learning on hard stages only, switched in for the last three hours: it solved more hard stages but cost action efficiency (successful runs longer than the human baseline 16% → 26%) [forum, not verified]. Both disclosed top configurations put memory per game first; we measured memory levers one at a time at ten or twenty games and spent most of our effort on the harness and training [inferred].
1A.2 Model and its training
Choice of model. The one model change that raised our battle score was someone else's: Flash-Next in keithtyser's kernel, 1.61 → 3.36 in battle (2.97 mean of three battles of the unchanged base), 3.02 → 9.43 on the public games [measured]. Franzen's packaging and draft then arrived inside the copy, their share of its jump not separated [inferred]. Alternatives we measured (§8, §11): gpt-oss-120b, untrained, cleared no level on 25 games (0 against 3.32 at 30 minutes, one run); DeepSeek-V4-Flash at 2–3 bits cleared 1 level against 3–5 in each of three working configurations; Gemini-3.6-Flash over an API — not servable in the kernel — cleared 10 levels on four games in ~14–15 minutes, when billing ran out, against 4 / 5 / 3 at 30 minutes; a REAP-448 pruned packaging cleared 26 levels against 18 on a rented card (with a larger cache, 16 against 9 GiB) and scored 4.00 in battle against 4.30 [measured].
August–September. Students distilled from Gemini teacher play cleared 0–1 levels in three tests (8, 16 and 20 games), and on 12 games 2 levels against 4 for their base [measured]. A usable teacher example costs $0.077 (§10), and LoRA on bf16 Flash-Next ran at 60 tokens/s, $519 per epoch [measured]; both were stopped as not worth their cost. Delivering changed weights (§9): a runtime adapter needs four serving concessions; overlaying the trainable bf16 shards and symlinking the rest costs nothing at runtime [measured] — our most reusable model result.
October, on the copy. Live test-time training cleared 20 levels against 70–78 in an hour (reasoning ×2.3) [measured]. Pruning 25% of experts: NLL +0.7% on recorded text; throughput +1.7–2.4% at equal concurrency (Kaggle probe), and +10.6% at 20 streams against the full model at 10 on a rented card, with half the generation per game; battle 22.65 with 20 concurrent games [measured]. The MTP draft: fine-tuning its first position (09.10) raised acceptance 2.72 → 2.80 (+2.9%) and throughput 639 → 652 tokens/s (+2.0%) on a rented card, one run per arm. The three-step chain version (10.10), which is the one in every battle build since 12.10, raised per-position guess rates 83.4 / 76.9 / 71.3% → 83.4 / 78.4 / 74.2% (first position unchanged) and reached acceptance 2.81 on Kaggle with no same-machine control; its battles, 26.44 / 24.09, are inside the copy cloud [measured] (§13B). The +2.9% / +2.0% belong to the first-position variant, not to the chain in battle.
Training on one card. With a vectorised sparse-attention indexer, LoRA on the served model trained at 719–807 tokens/s, ≈ $6 per 7M-token epoch [measured] (§13D). The first adapters (16–18.10) were trained on a path that differed from the served model by two silent loading errors — a randomly initialised 95 GiB n-gram table and a dropped GPTQ zero-point; with the table fixed the training-versus-server divergence was still a median 0.81 nats, with both fixed 0.07–0.10, the server's own noise [measured; Fig. f09; §13E]. Verdicts on those adapters are invalid.
Offline better, play worse. A reward-selected self-distillation adapter (pilot at 61% of an epoch: held-out loss −17.5%, first-move agreement 25.2 → 35.7%, p = 0.003; full epoch, the battle adapter: agreement 23.4 → 38.6%) cleared 7 of 36 held-out level starts against 3 (p = 0.31), then scored 15.73 in battle (20.10) against 24.36–30.04 (mean 27.23) for the same build without it; on a one-hour stand it cleared 54 levels against 80 at 2.2M tokens (games 3:14, p = 0.013; the control a different build) [measured]. A world-model adapter wrote correct simulators in 1 of 192 attempts (24 tasks × 8) against 9 for the base [measured]. Two adapters cross-trained at battle format lowered held-out loss 13–14% and cleared 13 and 6 levels against 25 for the base in 60 minutes [measured, one run per arm]. The review of 22.10 found that "move agreement" measured readiness to move — when both moved, base and adapter matched the successful move equally often (38 against 40 of 51; 34 against 36 of 52) — and three more silent data defects [measured]. Of five adapters trained after the two loading fixes three hurt play and two were indistinguishable [measured; Fig. f18]. Training for battle was closed on 22.10.
RL in miniature (§13G; gpt-oss-20b, one Kaggle GPU) first learned board-blind shortcuts; with action numbers permuted, unseen games rose 22.6 → 27.3% (p = 0.02, one seed, 0.4 short of the pre-registered +5), for reference, repeating the level's last move clears 58% of these starts, where the clearing move often repeats the previous one [measured]. It is not the battle model. A second step order (seed 2, 31.10, 63 steps) gave +2.2 points on the same unseen games (21.9 → 24.1%; levels 20:11, p = 0.16; games 6:1) [measured]: positive, not significant, half the first seed. A third (seed 3, 58 steps) gave +2.7 (23.3 → 26.0%; levels 21:10, p = 0.07) [measured]. Three of three seeds moved the same way, mean +3.2 points (2.2–4.6), significant in one; by the rule fixed before the repeats this is "not settled", below the pre-registered +5 [derived]. All three trained from scratch on one split of 16 games for ~60 steps; continuing an adapter was started (from seed 1 at step 64) but stopped by quota at step 34; that adapter, scored twice without training on the same unseen games, gave 26.0% against 26.8% for its starting point over four scores (−0.8 points; levels 17:21, p = 0.67; the starting point against itself −1.0, p = 0.61) [measured]. So 34 more steps did not move transfer; whether a full continuation helps is open (in two seeds the gain came after step 40) [measured; open question].
What holds. Offline metrics did not predict play: five adapters, three independent setups, one direction [measured; mechanism inferred]; serving a LoRA over the quantised MoE is not the cause [measured]. Not resolved: training on the full battle input (median 74k tokens, ~25 images; we trained on ≤ ~9.5–12k tokens); the 69 decision-point pairs collected on 24.10, never trained; gpt-oss-120b as a base for training (its 0/25 was untrained, one run); the chain draft against the original on one machine.
1A.3 Harness
What in the battle harness is not ours. From Duck: the Python tool loop and sandbox, frame segmentation, the model's past reasoning kept in history. From Franzen's ~500 KB patch: history to ~128k trimmed in one block, a priority scheduler fitted on 725 level attempts, an action named UNDO, animation frames, a 10× board image, structured world-model sections off, a 12,288-token answer cap. Ours in battle: a brevity instruction (from 14.10), a keep-alive pulse with a longer server wait (from 21.10), and in later builds a goal note (22.10) and a goal field with a server watchdog (23–27.10).
September, 32k window, starved server. About twenty layers; at one run a paired bootstrap over 25 games resolves ≥ 4 points, and the only layers outside zero were harmful [measured] (§2). Removing the model's past reasoning from history (the current turn's kept) scored 2.91 against 9.43, with the chain visible in the counters: prompt −17%, generation per answer +50%, queue 121.6 → 180 s, requests −34%, moves per game 175 → 81, levels 40 → 19; bootstrap −6.5 [−11.3, −2.7] [measured, one run; Fig. f19]. Tufa's own AVO layer (plan / implement / evaluate phases plus a supervisor) on our base scored 5.12 (games 3:14, p = 0.013); doubling calls per game moved levels 40 → 41 [measured]. Scott Le Grand's audit of what reaches the model found defects our layers had sat on: our harness lost 11–26% of world-model blocks (his 3%), and in 75–80% of move calls print output displaced the move result; his harness scored 8.01, 4.10 and 6.18 in one-hour runs against 5.00 for ours, and 4.11 in battle against 3.05–3.17 [measured].
October, on the copy. Macro moves: 51.61 against long-stand controls 46.06 / 48.73 (+4.2, indistinguishable), 24.85 in battle. A false-"no change" fix: p = 0.29. A history summary: 29.95 against 42.8–52.0 (4 games better, 21 worse), two variables changed at once. Brevity: −19% tokens per cleared level in a paired test (p = 0.035), within 0.5% on a repeat (p = 0.58); in battle 27.23 (four) against 25.27 (two) for the same copy with the draft alone and 26.61 (four) for the plain copy. The goal note: stated goals in 3.3% → 96% of calls (Fig. f12), 99 against 97 levels on paired stands (97.5 against 96.7 at equal tokens; Fig. f07), 24.01 in four battles. Testing an assumption after "impossible": 20/80 against 19/80; lower reasoning effort: 8/82 against 12/82 (p = 0.58) [all measured].
The pulse. The harness sends RESET to one game every 480 s until the server answers, so that a reported idle rule (no move for 15 minutes ends the run) cannot fire while weights load [tested on the real harness with a fake server]. Thirteen battles carried it (21.10–02.11); one returned 0.00 (22.10), the eleven after it none; before it, 2 zeros in 20 battles [measured] — not significant (Fisher p = 1.0, recomputed), and pulse, longer wait and watchdog cannot be separated without battle logs [inferred]. Its effect on the score is indistinguishable at five and four battles: v3 against v1 of the same build, Welch p = 0.35 [measured].
The audit (28.10, §13H) found exact move accounting, past reasoning at 42% of input, 34 of 69 history trims cutting into the current level, 52% of calls without a move [measured]; a decode analysis on 30.10 put recomputation after the harness's own trims at 6.0–6.7% of server time. The battle build was not changed.
What holds. Past reasoning is the model's working memory — our strongest harness result, measured twice: 2.91 vs 9.43 in the 32k regime (07.09) and, on the battle stack (131k window, Franzen's server, 25 games × 90 min, 31.10), 6.33 vs 38.47 with past reasoning removed from history (34 vs 93 levels; 0 of 22 scoring games better; paired bootstrap Δ −32.1, 95% [−44.1; −20.7]) [measured, one pair]. Removal cut the prompt 71.4k → 38.8k tokens (median) and left generated tokens unchanged (3.11 vs 3.18 M), yet answers grew 1,049 → 1,391 tokens (median), calls fell 2,003 → 1,681 and moves 4,494 → 2,762 [measured]: the model re-derives what it had already worked out. Added text did not move the score at any resolution we had; imposed structure and stall triggers hurt [measured, single runs]. About half of all calls make no move under every harness we read (Duck 50%, Franzen 52%, face-of-agi 52% by their count) against 10% for a stronger teacher [measured]. The model finds the goal (40/40 successful, 36/40 failed attempts) and fails on mechanics (26 of 40 failures) [measured; Fig. f11]; failed attempts often abandon a correct goal after an untested "impossible" (7–8 of 15; Fig. f10) [measured]. Not resolved: the goal note; the image scale (4× +25% tokens in one twin run, inside the instrument's noise; 8× never run); the zero-risk switches from the audit.
1A.4 Server and its performance
September. The inherited vLLM profile gave the KV cache 5 GiB, about 105k tokens — "maximum concurrency 3.21×" at a 32k window; the log showed Running: 3, Waiting: 22 all month, a request spent 84–87% of its time queued, and throughput was about 900k tokens per hour, ≈ 250 tokens/s [measured]. We read this for three weeks as a compute limit (§11.1); fp8 KV, tested at the unchanged 32k window, raised capacity ×1.32 and levels not at all — the wrong question, since its use is to afford a longer window [measured].
The disclosed stack (from 01.10): fp8 KV in a 1.01M-token pool, ten streams, a ~128–139k window, 93–94% of prompt tokens from the prefix cache, a draft accepting 2.60–2.75 tokens per step, a median 723 tokens/s [measured] — about 2.5–3× September's throughput [inferred, mixed measures]. Battle 4.30 → 24.25 came with its harness and packaging; the server's share is not separated [inferred].
Instruments we built (§13). A replay bench of recorded requests reproduces itself to ~0.3%; it showed twenty games dropping to 415 tokens/s (−32%) as histories are evicted, a host-RAM tier repairing that (650, +7%), and a silent cap of 12 concurrent requests [measured]. A KV-quality probe on 20 recorded contexts put fp8 against BF16 at +0.0026 NLL. Our NVFP4-KV patch for the sparse-attention path enlarged the pool ×1.50 at +0.0015 NLL; its battle (10.10) returned 0.00, cause not determined [measured].
The score follows total generation. On the game bench of 03.10, twenty games on NVFP4 KV without a host tier scored 21.79 and a 250k window with four concurrent games 24.69, against controls of 50.63 / 53.06 (22 of 25 games lost, p = 0.0002), while levels at an equal number of moves were unchanged (61/61, 62/68) [measured]. A different packaging of the same model (RadixArk NVFP4) left a pool of 656k tokens, ran at 503 against 674 tokens/s and scored 23.27 [measured]. Adding generation to stalled games yields little (§13B) [inferred].
Matrix and profile (04.10, a rented card ~1.6× slower than Kaggle's in CPU loop time). Zero to three draft steps: 438 / 524 / 590 / 599 tokens/s, +37% from the draft; four steps do not start; eleven or twelve slots +1.5%; the GPU idles 16–21% of each decode step [measured]. Our additions, in tokens/s: first-position draft fine-tune +2.0%; a hot-token draft vocabulary −4.8%; test-time training inside the server −14%; 25% expert pruning +1.7–2.4% at equal load (+10.6% at 20 streams against the full model at 10; battle 22.65). Face-of-agi's p37 build ran at ~1,400–1,525 against our copy's 570–640 tokens/s but cleared 40 levels against 66 at equal tokens per game (p = 0.04) and scored 19.52 [measured].
The battle build under load (24.10, two 132-minute stands): 628 / 607 tokens/s; 8.9 / 8.5 of 10 slots busy; the pool above 0.9 full only 2.1% / 1.6% of the time, no evictions; a request spends 97.2% of its life in decode [measured]. A battle generates about 18–19M tokens, ~90 calls per game [inferred from stands]. Decode on 30.10 (Fig. f17): median throughput by number of running requests 1 → 161, 4 → 507, 7 → 691, 10 → 796 tokens/s — rows of one stand's log grouped by load, an observation, not a controlled experiment [measured]; the step grows from 16.0 to 33.9 ms, about 14.5 + 1.9 ms per request [inferred]; a stream's speed does not depend on its history length (70.8–74.3 tokens/s at 0–100k) [measured]. Decode is bound by reading weights [inferred from the 04.10 profile and from stream speed not depending on history length]. Prefill stalls decode for at most 18–19% of the window, of which recomputation after the harness's own trims is 6.0–6.7% of the window [measured; upper bound inferred].
Pending. SGLang 0.5.22 against Franzen's 0.5.19 build: our port first found a reference cycle that held source weights during repacking (load OOM at 94.34 of 94.97 GiB; fixed, 69.91 GB) and a failure at prefill graph capture on fp16 expert scales; the first full stand (31.10) was 22% slower per request (A/B 0.779, 95% [0.71; 0.85]) with prefix-cache hits 63% against 83%, because 0.5.22 lacks Franzen's own two linear-state cache patches [measured; cause inferred from the server logs]. With the two patches ported (about three lines each) the cache matched (82.2 vs 82.3%) and speed was at parity: A/B 1.023, 95% [0.975; 1.071], 129 pairs, 410 vs 410 generated tokens/s [measured]. The newer runtime brought no gain on our load; the build's value was in its owner's patches. MXFP8 (online FP8 for the dense layers, present in the fork but off; expected +5…15% [assumed]): the stand was lost — our own likelihood probe on the same server crashed it before the comparison; that run's log shows decode medians of 549 against 703–766 tokens/s at ten requests, but on different machines, so this is not settled [measured; comparison across sessions]. Where a decode step goes on the Kaggle card (stand v5, Franzen's server, ten streams): nvidia-smi shows the card busy 88.8% on average (median 99%), at 360–390 W of a 600 W limit and a steady 2.42 GHz clock — no power or thermal limit [measured]. A profiler trace at nine requests: the verify step is 26.2 ms of GPU work — Marlin experts 46%, dense BF16 matrix products 31%, sparse attention 8.5%, linear-attention state 3% — and with the draft and state-recovery graphs about 30.5 ms per ~30 ms step, so on Kaggle the card is busy nearly all the time [measured; occupancy inferred]. The 16–21% idle of 04.10 was a pod with a slower CPU; on Kaggle a C++ host loop has little left to win [inferred]. Franzen's small-batch kernel stops at M ≤ 32, while the verify step has M = 36–40, so the dense products fall back to cuBLAS [measured]. A microbenchmark on the battle card (01.11; no model; the battle's torch, Triton and cuBLAS 13.1; under a CUDA graph with cold weights; measured read peak 1,520 GB/s) shows this costs little: the ten dense shapes of a verify step take 6.85 / 6.83 ms per step at M = 36 / 40 with cuBLAS, 8.17 ms with Franzen's kernel as shipped, and 6.62 / 6.61 ms with the best of 32 tilings per shape — 0.23 ms, under 1% of a 30.5 ms step [measured]. The large projections already stream at 78–98% of the measured peak; the loss is in the small matrices (hyper-connection mixing, shared expert, router: 26–62%), which are launch-bound, so retiling does not help and only fusing them could (at most ~1.3 ms, ~4%) [measured; bound inferred]. Below M = 32, where the kernel is active, it is slower than cuBLAS on four small shapes and faster on five others, 0.3–0.6 ms per step slower in total; choosing per shape would save 0.7–1.0 ms against the kernel as shipped [measured; microbenchmark, not the live server]. Earlier estimates of +9% from retiling and +15–18% from FP8 weights assumed the 58% average was recoverable. Measured instead (01.11, same card, same method): storing the four large projections and the lm_head as FP8 with one scale per row and reading them with a tuned Triton kernel saves 1.71 / 1.61 ms per verify step at M = 36 / 40 and 2.0–2.2 ms at M ≤ 32, about 5–6% of a step; the kernel matches its own FP8 reference to 0.2%, but only with a per-shape tiling table (the default tiling saves 0.5 ms and is slower than BF16 on the output projection) [measured]. FP8 rounding itself moves a projection's output by 2.65% (relative norm, random weights) [measured]; prefill, which must expand the weights, gets 7–32% slower per layer [measured], about −0.5% overall [inferred]. Net, about +4% tokens, worth ~+1 point against a ±5.9 battle spread [inferred].
What holds. September's ceiling was the KV cache, not compute; at ten games the disclosed stack's cache does not bind, above twelve without a host tier it breaks; fp8 and NVFP4 KV cost nothing measurable in likelihood (20 contexts); the draft gives +37% and is exhausted at three steps [measured]. The one whole configuration that transferred was Franzen's; transplanted components did not, nor did a whole copy of face-of-agi's build (19.52) [inferred from single battles]. A server change carries risk no public run exercises [inferred]. If the score follows generation in proportion, +10% tokens is worth about +2.6 points, below one battle's ±5.9 [inferred]. Not resolved: twenty games with a host tier in battle (a bench tie, never submitted); the cause of the NVFP4-KV zero.
1A.5 The Kaggle platform
Machine and rules. AMD EPYC 9B45, 176 GB RAM, one RTX PRO 6000 (96 GB) at 600 W [measured]; one submission per day; 110 hidden games in one run; a 9-hour limit that, by Tufa's account, includes about 40 minutes before the notebook starts [reported, not verified]. Session limits, queues of 1–3.5 hours, push errors, two mount paths for the data and a network outage cost days of development; we found no points lost to them [measured].
Weight loading. Reading the same 181–185 GB took 254–4,412 s (4.2–73.5 min), and the server became ready 10 to about 78 minutes after notebook start [measured]. 95% of that variance is weight reading, which is bimodal (450–715 against 40–220 MB/s); loads slower than 700 s were 3 of 39 before 17.10 and 11 of 29 from 17.10 (13 counting 500–700 s; on the wider log set of Fig. f08, 3 of 50 and 9 of 36) [measured]. The strongest correlate is the start hour: loads started at 00–01 UTC were slow in 7 of 8 cases, in other hours about one in seven (7 of 60; 7–9 of 49–51 on a recount) [measured, observational; Fig. f16]. Our battles were submitted at 00:05–00:31 UTC (24 of 30 in October). A second load in the same session was always slow (1,288–2,365 s), so a server restart mid-battle costs 20–40 minutes [measured]. A forum claim that slow-loading machines also generate ~10% slower is not confirmed on our runs (777 and 783 against ~775 tokens/s) [measured; forum, not verified]. Battle scores of the family and the goal-field build by submission hour do not differ (00 UTC: 15 battles, 25.62 ± 3.28; 03 UTC: 4 battles, 25.73 ± 1.13; Mann–Whitney p = 0.80), but submission is not the run's start [measured].
Zero-score runs. 10.10 (our NVFP4 KV), 17.10 (a structured note; its phase-A machine loaded for 847 s at 00 UTC and phase A ended before the server answered) and 22.10 (a goal-note build that had the pulse) all returned exactly 3,411 bytes, against 3,792–4,066 for scored runs — no level cleared in any game, not a weak run [measured; Fig. f14]. Kaggle exposes no logs of the scored rerun. Candidate causes — the idle rule (weakened for 22.10 by the pulse), a slow disk, a missing KAGGLE_IS_COMPETITION_RERUN variable, which another participant reports for 22.10 and which is the only signal our notebook uses to detect the scored rerun — are not resolved.
The 29–30.10 decline. The pulse build scored 30.85, 22.12 and 21.58 on consecutive days; the two lowest of fifteen family battles are the two most recent (p ≈ 0.01, a test chosen after seeing the data), while over nineteen battles there is no trend (Spearman ρ = −0.23, p = 0.35) [measured]. Nothing changed on our side: the same notebook version, its code dataset untouched since 18.10 [measured]. Two battles cannot separate a platform decline from a streak of slow machines [inferred]. The next two battles of the same build scored 22.92 (31.10) and 30.97 (01.11, our best): seven battles of this build span 21.58–30.97, mean 25.60, SD 3.98, so the dip sits inside its own spread [measured; inferred].
The price of loading. A timeline simulation of 110 games from recorded runs puts 30 → 75 minutes of loading at about −3 battle points (±1–2) [assumed]; a proportional estimate gives −1.5 to −2.5 for 30–40 minutes of 532 [inferred]. The first final battle, submitted at 04:00 UTC instead of 00:05, scored 30.97 [measured]; one battle per day does not resolve the hour effect [inferred]. The second, at 03:30 UTC, also changed the build (FP8, §1A.1) and says nothing about the hour.
1A.6 Measurement
| Instrument | Spread of identical runs | What one comparison resolves |
|---|---|---|
| Battle, one build family (15 battles, 01–30.10) | SD 2.85 on a mean of 26.08 | one battle ±5.9; three ±3.6 [inferred, 2·SD·√(1/k + 1/15)] |
| 132-minute public run, September (7 repeats) | 4.45–9.43, SD ≈ 1.6 | one pair ≥ 3.9; five pairs ≥ 1.75 [inferred] |
| Long stand, 25 games, 2 h (October; the second pair 100 min) | 46.06 / 48.73; one build 52.02 / 42.77 | one stand ≳ 12 points [inferred] |
| Paired level-start test, 16 levels × 5 | 15–30% between identical runs | effects ≳ 15–30% in tokens per clear |
| Public set → battle (September) | ratio 0.28–0.53 | a doubling on a rented card became 4.00 vs 4.30 |
What we did. We measured the spread of identical runs before trusting a difference (§1): seven repeats of one September build, eight long stands of one October family (Fig. f05), fifteen battles (Fig. f04), and a paired level-start test whose two controls differed by 30% in tokens per clear [measured] (§12, §13D, §13H). We wrote decision rules before runs — "≥ 20 confirms the base" (01.10; then 24.25), a +5 threshold for a breakthrough (05.10), "≥ +5 points on unseen games at p < 0.05" for RL (29.10; then +4.6 at p = 0.02, recorded as short of the rule). We compared runs at an equal number of generated tokens, as the disclosed teams do, and computed counterfactuals offline on a deterministic engine replica (561 of 561 moves reproduced): perfect efficiency on cleared levels is worth +24% and one more level per game +137% (§7); on October stands inefficiency costs 2.5–3.3 points in 100 (Fig. f03); scoring each game by its best of eight stands raises 45.7 to 67.5 (Fig. f06) [measured].
What skipping it cost. The most expensive errors of the season were measurement errors [inferred]. A score formula valid only for one-level agents (01–02.09) cost about nine days on the wrong quantity. A cache limit read as a compute limit (10.09) closed the serving layer for three weeks. "The layer pays on strong games" was regression to the mean: unchanged code lost 19.8–20.0 points on the same four games. Two of the author's measured choices were reverted on bench differences inside bench noise (04–05.10); one stand pair showed a false "−12" for a fix later indistinguishable (11.10); brevity was called "the single positive result" (§13C) before its repeat failed; a history summary and gpt-oss-120b were closed on single, weak runs. Development runs from different days are not comparable, because the game clock starts before the server is ready: one control lost 41 of 132 minutes to loading [measured].
What holds. One battle sees about ±5.9 points against fifteen, and none of our October additions exceeded it upward, nor the ±3.6 of three battles; our trained adapter fell 10.4 points below the family mean (15.73) [inferred from measured battles]. Stands do not transfer to battle reliably [measured]. The score budget is bounded on efficiency and open on depth [measured]. Not resolved: stand-to-battle transfer in October (stands ≈ 46–52, battles ≈ 26; the stand score's formula was not reconciled with the official one); the twin test's noise on one machine (its two controls ran on different machines); and every account of a zero or a low battle, since battle logs do not exist.
2. Twenty layers, no signal
Layers, each with pre-registered thresholds and a paired bootstrap: goal gate with refutation, stuck reset, scripted explorer, level-fact injection, macro-actions, prefix caching, best-of-N voting, ten executable-world-model variants, history truncation, oracle rule injection, doubled call budget, world-model carry, three search layers. No delta fell outside baseline spread in the helpful direction: no bootstrap interval excluded zero upward, and two of these layers had intervals below zero (oracle rule injection −4.2 [−9.4, −0.3]; a search tail −1.51 [−3.17, −0.26]). Three findings run against intuition:
* Doubling calls per game (56 → 112) moved 40 → 41 levels: the extra budget went into revisiting states (returns ×2.5). * Removing the model's past reasoning from the history (the current turn's reasoning kept) destroys the agent (2.91 vs 9.43), while adding input information — oracle rules, world-model digests, level facts — changes almost nothing: the model restated an injected rule in 27 of 27 answers and still did not act on it. * History is working memory: cutting it to two turns ("amnesia"; a cloud bench of 9 stalled games, 27 calls) halved the share of calls with a move (44% → 22%, p = 0.109), and a 16k window cleared 5 levels against 19 in one run on a rented card.
In hindsight the third finding was the one to follow, and we followed it in the wrong direction only: we tested a shorter window (16k), saw harm, and never tested a longer one. The second finding also admits a reading we could not separate from ours at the time. We concluded that the model cannot use injected information. Under a capped window that is trimmed to fit, every injected line also displaces a line of the agent's own recent history; rellik13 reports two score drops from added prompt text under a ~37k cap (6.42 → 4.16 and 13.40 → 9.98) and attributes them to exactly this. Our data cannot tell the two readings apart, because we never ran an injection layer with history held constant.
3. Mechanics without goals
The same model that gains no score writes correct dynamics: its synthesised transition programs predicted unseen transitions in 86% of changed states (89/103), and on one game a counterfactual check was correct in 200 of 200 out-of-distribution clicks. Asked explicitly for the level goal, it was right in 1 of 6 games, and a learned goal detector lost to the untrained heuristic "largest board change" in all 5 held-out games. Six transfer measurements (imitation and cross-game policies, few-shot, online context, shuffled labels, learned detector) returned zero transfer to held-out games at 0.64–0.68 accuracy on training games — memorisation, not rules.
4. Goals as verifiable predicates
We define a level goal as a predicate over board state drawn from a 21-template vocabulary (colour presence and counts; component counts; single component; rectangle; symmetry; shape match; containment; region-pattern equality; match to a hole — the "complementary object"; match to a shape seen earlier; and four templates comparing the state to the level's start). Induction keeps predicates true at the completion frame and false at every state seen on that level.
* Separation, honestly controlled. The true completion state is separated from every other state of its level on 44 of 53 levels (83%). When a random non-goal state is declared the goal and the same search re-fits its parameters, it is separated in 536 of 1060 trials (51%). Our first control (8%) re-used parameters fitted to the true goal and so tested the wrong hypothesis: the honest gap is 83 against 51, not 90 against 8. * Cross-level transfer, leave-one-game-out (26 pairs; per-kind rates estimated on the other 14 games): Coverage@1 = 27% for all three parameter-transfer rules, Coverage@3 = 35% for the monotone rule ("fewer than before"). Nulls: the same transfer from a pseudo-goal covers 1–2%, and from another game's goal 0–1% — so what transfers is the goal, but weakly. * Per-kind precision curates the vocabulary: "colour c is gone" transfers 6/6, "exactly m components of colour c" 4/34, shape matches 11–12%. Keeping kinds above 40% raises precision from 31% to 70% and halves coverage (23 of 53 levels vs 38).
5. Search guided by goal hypotheses
A best-first search over the real engine, ordered by a dense residual (assignment-problem matching between object sets: shape mismatch plus centroid distance) toward parameter-free candidates from the same vocabulary, solves 14 of 25 first levels — as many as blind BFS under identical budgets — using 64,092 moves against 127,911 on the 13 games both solve. One design fact was measured, not assumed: step-function residuals carry no signal (median Spearman ρ = −0.02) while dense matching residuals do (ρ = 0.20–0.93 where the goal is in the vocabulary).
Search also bounds the popular "learn a simulator, plan inside it, execute the plan" recipe. Granted a perfect simulator — search on the real engine with state snapshots — and 420 s and 20,000 states per game, breadth-first search finishes 1 of 25 games (tu93: 9 levels in 185 moves against a 462-move baseline, a perfect level score, after 8,334 search moves), clears at least one level in 15, and none in 10. What limits planning here is the size of the space, not the fidelity of the model of it.
6. Why search does not pay, and what the goal hint did
A level scores (baseline_actions / actions_used)², and search actions are counted: a level reached after thousands of search moves is worth ≈0, and a level the model would have solved itself is destroyed if search touched it. Counterfactual replay of three base runs quantified this before compute was spent — search after 30 minutes of no progress costs −0.2 to −1.1, before the model's first call −2.2 to −2.3 (30–42% of the score) — and a probe confirmed sign and magnitude (−0.58 to −2.78).
Checked in the live bundle rather than assumed: the harness sets ONLY_RESET_LEVELS=true process-wide, so the engine never performs the full reset that would open a fresh scored run, and the scorecard admits one run per game id. The "win, then replay optimally" strategy that works offline is unavailable in play, so exploration cost is never recovered — which is why a training-free graph-exploration agent ranked 3rd by levels solved would score ≈0 here.
The remaining zero-cost use is to hand the model the verified goal of the level it just completed. Two probes (next-level rate L2|L1): 0.25 vs 0.35 baseline with a noisy vocabulary, 0.38 vs 0.28 with the curated one (p = 0.51); whole-run scores were at the bottom of the control range in the first (5.61 against 7.23 / 6.32 / 5.46) and below it in the second (2.69 against 6.11 / 5.40 / 3.91, with first levels 13 against 19 / 19 / 16). The model reads the hint (112 of 379 responses), but the effect is indistinguishable at this sample size, and whole-run score cannot measure it: half the score comes from first levels the layer never sees.
The decisive test is about progress, not recognition. For every transition on a known solution path we computed an object-matching distance to the goal and asked whether it falls more often for the winning action than for its siblings. With the true completion frame as the goal, it does: 0.69 against 0.49, AUC 0.629 over 89 transitions in 10 games. With the goal transferred from the previous level it does not: 0.17 against 0.09, AUC 0.473. A pre-registered criterion (AUC ≥ 0.60 and a ≥ 0.10 gap) therefore closes this line as a score lever while leaving it intact as a measurement instrument.
7. The score budget: where points can come from at all
Before asking which layer to build, we computed what the scorer can pay for. Using the rule in the dataset description (the ratio is clipped at 1.0 before squaring, so a level is worth at most 100; the installed official client library caps at 115 instead, and which rule the Kaggle scorer applies we did not verify; each comparison below stays within one rule) on three full runs of the base:
| Counterfactual | Score | Change |
|---|---|---|
| As played | 8.11 | — |
| Every completed level played perfectly | 10.06 | +24% |
| At least one level cleared in every game (16% of game-runs clear none) | 10.68 | +32% |
| One more level cleared per game | 19.21 | +137% |
The first row of that table is the important one: 64% of the 121 completed levels are already at the cap (median ratio baseline/actions = 1.16), and the levels below the cap average 47 of 100. Efficiency work is therefore bounded at about a quarter, while depth is worth more than the entire current score. Every harness lever we measured acts on the bounded half.
Two further counterfactuals close popular ideas. First, the competition scores a game by the best run, and our run-to-run spread is large (ar25 scored 24.5 / 8.3 / 22.2 across identical runs), which suggests two short passes instead of one long one. Replaying the three runs truncated to half the per-game cap and taking the per-game maximum of two gives 6.56 against 8.11 — worse, because halving the time costs 38% and the maximum recovers only part of it. Second, the marginal value of time falls steeply: the second half of a 132-minute game produces 38% of its score, while a further 108 minutes (a 4-hour cap probe) adds 24% more levels but only +3.8% score. Reallocating time between games is therefore bounded by roughly the same +4%.
Both bounds need their scope stated, because Milestone 2 tested them from outside. They are computed on runs in the starved serving regime of §1 with a fixed cap per game, and they bound one intervention — more wall-clock for a game whose history is trimmed to 32k on every request. They say nothing about deciding which game holds a server slot at each moment: the best disclosed solution opens all games at once, keeps ten active and re-queues a game by a priority built from its level, its remaining levels and the moves and tokens already spent without progress (§12). Its author did not isolate the scheduler's contribution, so it neither confirms nor refutes our +4%; what it does refute is our use of that number to close the topic. The depth row of the table stands, and it is where the disclosed solutions gained: they moved the lever our layers could not, by giving the agent memory rather than instructions.
8. Is the ceiling the model?
Twenty negative layers make the claim scaffolding cannot create ability plausible but not tested: we never varied the ability. We did so twice, holding the harness, the engine and the games fixed and changing only the core.
A stronger model, crushed by quantisation, is worse. DeepSeek-V4-Flash is a larger and stronger model than our baseline, but the competition GPU (96 GB) only fits it at 2 or 3 bits. On the same four games at a 30-minute cap it cleared 1 level against 3–5 for the baseline, with and without thinking, at both quantisations (three working configurations, 1 level each; the 3-bit run with thinking was on eight games, 1 against 5 / 5 / 5; a fourth, 2-bit with a 1,500-token answer cap, cleared 0 with its parsing layer failing — 3,120 parse errors). Server timings attribute the loss to generation, not to prompt reading: 18.7–27.5 tokens/s per request, with prompt processing 9–13% of wall time. Capacity that does not fit the card is not capacity.
An uncompressed stronger model is much better. Gemini-3.6-Flash through the same harness on the same local engine cleared 10 levels on four games in ~14–15 minutes, when billing ran out, against 4 / 5 / 3 for three baseline runs at 30 minutes. Per-level move efficiency (human baseline ÷ agent moves) had median 0.82, with 8 of 10 levels inside 2× the human baseline; the baseline's own median is 2.16 on the levels it does clear, so the teacher trades efficiency for depth. Repeating one game at the battle context window (32 768 rather than the 131 072 we first used by mistake) reproduced the depth exactly — 4 of 9 levels on tu93, RHAE 17.9 against 15.1 — showing the advantage is not an artefact of extra memory.
What the stronger model does differently per turn. On the same four games, counted from transcripts: the teacher emits 45 lines of tool code per harness turn against the baseline's 14, executes 6.2 engine moves per turn against 2.1, and leaves only 10% of turns without any move against the baseline's 50%. Both consult the frame segmentation at the same rate (0.48 vs 0.57), so the difference is not tool access but what is done with one look. Note that this does not identify spare turns as the binding constraint: doubling the call budget in §2 moved 40 levels to 41. The teacher is not taking more turns, it is finishing more per turn.
At the time we drew the conclusion that these two swaps, together with §2, bound the problem: the scaffolding ceiling is set by the core model, and the core model that fits the hardware budget is the constraint. That conclusion was wrong, and Milestone 2 is the counterexample (§12): the same model on the same card scores 20–28 where we scored 4.30. What the two swaps actually show is narrower. A model crushed to 2–3 bits loses; an uncompressed stronger model wins; neither comparison varied what the baseline model was given to work with. We had varied the text put in front of the model twenty times and the model twice, and never the amount of its own history it could keep or the compute it received per game — the two quantities the disclosed solutions changed. A harness change that matters therefore has a third route besides buying ability or exploiting structure the model already has: stop taking away what the model already had.
Executable world models are a frontier-model tool. The most successful published recipe for this benchmark has the agent write an executable Python model of the game, verify it against every past observation, and plan through it before acting; it reaches 58% mean per-game RHAE with GPT-5.5 (15 of 25 games fully solved), and related systems report 88–100 with frontier models. The recipe fails on our model, and the failure is specific. The model does capture local dynamics — the changed cells of a transition (§3) — but it cannot reliably write an exact executable copy of a whole unfamiliar game: in an offline counterexample-guided probe it reproduced all observed transitions of a level in only 6 of 24 games, and a simulator that also predicted unseen transitions better than a trivial "nothing changes" baseline emerged in just 2; in live play, given validation and planning helpers, it never once called the validator (0 of 1,348 tool calls); ten earlier variants of the same loop (§2) showed no signal. Where it does succeed, the simulator is real rather than a fit: in tn36 and sp80 it predicted 11 and 10 of 16 unseen transitions against 0 for the "nothing changes" baseline, while in the other four "exact" games it did no better than that baseline or worse. This suggests a selective use rather than a general one — the harness lets the model plan through its simulator only after the simulator has reproduced every observation of the level without error, and otherwise leaves play unchanged. The upside is bounded, since saving moves on levels the agent already clears is worth at most about a quarter of the score (§7), and a perfect simulator alone does not name the goal (§5). For writing an exact simulator, the bottleneck is the model, not the sandbox it writes code in.
A world model need not be code written by the language model. We compared the forms it can take on the same recorded play, predicting each transition from the ones before it, as in a live game: a lookup table of observed transitions is right 78–90% of the time but only applies when the exact state recurs (2–8% of moves); a convolutional next-frame predictor reached a median of 0%; programs written by the model predicted about half of changed transitions in our earlier measurements but rarely a whole level exactly. Between these sits a form that needs no language model at all: the structure of the rules is fixed by us — an object moves in the direction of the action until blocked; cells a mover leaves reveal what was under it; a click recolours the object under it — while colours, step sizes, directions and passable cells are learned from the game's own transitions. Online, on 25 games, it predicted 24–27% of transitions exactly against 18–23% for "nothing changes", with large gains only where the mechanics fit its templates (ft09 65% vs 2%, sc25 58% vs 32%); most games use other mechanics (pattern rotation and recolouring, pushing other objects, shape changes). It is cheap and verifiable, but so far covers only simple mechanics. This should not be read as "only frontier models can play": the recipe has been demonstrated with frontier models and fails with ours, and the middle ground is untested. Per move, a stronger but non-frontier model plays no better than ours — at equal move counts our teacher (Gemini-3.6-Flash) cleared the same number of levels (e.g. 2 vs 2.6 on tu93 and 3 vs 3.0 on lp85 by move 100); its advantage was doing more per call (6.2 engine moves per turn against 2.1, and only 10% of turns without a move against our 50%). We therefore invert the division of labour: rather than asking the model to build a simulator, the harness does more of the observation work for it — it computes the objects on the board and what changed after each move, and places them directly in the model's context. The motivation is measured: about half of all model calls are spent only looking at the board through code (printing board fragments in 51–75% of them, object lists in 34–61%), each costing a full queued request. This layer was measured on two rented machines. On the second, about 1.5× slower than the one that produced its control (§11.6), it cleared 17 levels in 30 minutes against 26–32 for the control on the faster machine; that comparison is invalid. On the first, one card on 27.09 with its own controls, a first version scored 1.46 (10 levels; the mechanism did not engage) and a revised one 2.75 / 2.51 (16 / 15 levels) against controls of 2.60 / 2.98 / 1.45 (17 / 16 / 10): indistinguishable, with the mechanics shifted (95 inspections against ~140, answers +35%) [measured]. The verdict stays open. Two of the four disclosed Milestone 2 solutions, including the best, switch the harness's structured world-model sections off; its author reports that disabling them was the largest improvement in that area, because with a large rolling context the conversation itself retains the evidence. That is consistent with our ten null world-model variants and suggests the same displacement mechanism as in §2.
Three cheap tests of the remaining structure, all negative. Ranking the winning move against its siblings by displacement toward the true goal reaches top-1 0.44 (chance 0.19); with the goal transferred from the previous level — the only goal available in play — it is 0.19, exactly chance, and the measure is flat across all moves in 53% of states. Choosing moves to discriminate among surviving goal hypotheses (median 79 per state) reaches 0.23; averaging progress over all live hypotheses reaches 0.34. Transferring verified mechanics between levels has a hard score ceiling of +2.4% (112 of 2568 moves on cleared levels are first uses of a simple action, and 78–80 of them had no stable effect on the previous level anyway). Finally, stalled games are not looping: 72% of their moves reach states never seen before, so anti-loop machinery has nothing to catch — what repeats is the pattern of moves (64% of click triples), not the position.
Two measurement pitfalls worth stating. First, the OpenAI-compatible endpoint of our teacher reports completion_tokens without thinking tokens while billing them as output: on a fixed prompt it reported 528 completion tokens with 8 076 total, hiding 7 473 — a token-based cost model understates spend twofold. Second, a serving parameter transplanted between stacks silently changed the experiment: 131072 was the server-wide context split across four slots in one setting and the per-conversation window in another, giving the agent four times the memory it has in competition. Both were caught only by comparing against an external ledger (the account balance, the recorded competition environment), which we now do by default.
9. Delivering a modified model to the competition kernel
If the ceiling is the core model, a team's next step is to change it — which raises an engineering question we could not find documented anywhere: can a modified model actually be served inside the competition kernel, where there is no internet, the checkpoint is pinned by hash, and the card is already 89% full? We measured both routes on the real kernel. This section is the part of our work most directly reusable by others.
The checkpoint is a 125B mixture-of-experts model in 419 files: 68 GB of routed experts in NVFP4, 16 GB of bf16 modules (full attention, linear attention, routers, shared expert, embeddings) and 51 GB of per-layer embeddings that live in host RAM. Only the bf16 part is trainable in practice, and a rank-16 adapter over all 48 layers is 21.9M parameters — 0.018% of the model, 44 MB on disk.
Route A — serve the adapter at runtime. vLLM does apply LoRA to this architecture, and — contrary to a risk we had recorded from an earlier experiment — it applies it both to the 12 full-attention layers and to the 36 linear-attention layers; the only modules it silently ignores are the vision tower's. Getting the server to start, however, required four concessions, each found by a separate failed run: the KV cache had to drop from 5 GiB to 3 GiB (it cannot go below 1.55 GiB, the floor for a single 32,768-token window); the batch had to drop from 8,192 to 2,048 tokens, because with LoRA enabled vLLM materialises a logits buffer over the whole batch (8,192 × 248,320 vocabulary × 2 bytes = 3.79 GiB); CUDA graphs had to be disabled; and only one adapter may be active per batch. All four cost throughput, which on this task is the scarce resource.
Route B — merge into the weights and overlay the files. Because the trainable part is confined to four bf16 shards (0.32 / 1.27 / 3.68 / 10.73 GB), the trained delta can be written into those shards and the remaining 119 GB left untouched. The kernel builds a directory in which 419 model files are symbolic links to the mounted read-only checkpoint and only the modified shards are real files, rewrites the manifest, and points the loader at it. Weights must be patched in place at the byte level — re-serialising a safetensors file changes its size and fails the harness's byte-total check. Verified on the kernel: the harness accepted the overlay (files=419 bytes=135,253,622,894), the server started on the stock profile (5 GiB cache, concurrency 28, CUDA graphs on), and zeroing two matrices in 2 of 48 layers visibly degraded play — the model answered a fixed prompt with 1, 2, 3, where the stock model answers 1, 2, 3, 4,, and 25 games produced 73 moves instead of thousands. Route B therefore delivers modified weights at no runtime cost, and a team never needs to upload the 135 GB checkpoint: a 44 MB adapter travels as a dataset and is merged inside the kernel.
10. What a distillation attempt would cost, measured rather than guessed
For teams considering the obvious next step — distil a stronger teacher into the served model — we measured the inputs. A teacher (Gemini-3.6-Flash) on our harness produces one training example per harness turn, not per model call: at the battle context window a turn takes 1.74 calls, and 69 turns cost $2.78, i.e. $0.040 per turn. Only turns inside levels the teacher actually cleared are usable (52%), giving $0.077 per usable example; the median example is 26,830 tokens with the history reconstructed as the harness sends it, so an epoch over 2,500 examples is 67M tokens. Training the bf16 modules only — the fast regime, since adapters over fused expert tensors ran 12× slower in our earlier work — puts a two-epoch run in the low hundreds of dollars on rented B200s rather than the four figures we first estimated.
One accounting trap is worth publishing: the OpenAI-compatible endpoint of the teacher reports completion_tokens without thinking tokens while billing them as output. On a fixed prompt it reported 528 completion tokens against 8,076 total, hiding 7,473. A token-based cost model built on that field understates spend twofold; only the account balance is authoritative. Reducing the thinking budget does not help either: at reasoning_effort: low the per-call price fell 2.3× but the model compensated with 3.7 tool iterations per turn instead of 1.7, leaving the price per turn unchanged ($0.039 vs $0.040) and the play visibly worse (first level cleared in 12× the human baseline).
11. What September taught us: the serving stack was the binding constraint
Everything in sections 2–8 was measured on a serving configuration we inherited from the public line and never audited. At the end of September we audited it layer by layer, and each finding would have been available at the start of the month.
- The KV cache, not the answer length, capped throughput. The inherited profile gave vLLM 5 GiB of KV cache: 105k tokens, "maximum concurrency 3.21×" at a 32k context. The server log showed
Running: 3, Waiting: 22for the whole month and a request spent 99 s of its 117 s in the queue. Our earlier conclusion that "the server computes about 3.2 requests at a time, so the lever is the answer length" (10.09) was this cache limit misread; it closed the serving layer for three weeks. With 8 GiB on a rented pod, requests per 30 minutes rose from 331–376 to 456 and moves per game from 21 to 30. - The runtime was a month old. The pinned vLLM build dated from 26.08; vLLM 0.30 (22.09) loads the same weights 2.4 GiB lighter (79.43 vs 81.8 GiB), which is exactly the headroom that had made 7–8 GiB of cache fail on Kaggle. It added no speed of its own: 0.30 with 9 GiB matched the old build with 8 GiB (450 vs 456 requests, 18 vs 19 levels).
- Model packaging is a lever, and we compared none. A REAP-pruned variant (448 of 512 experts) weighs 69.2 GiB on the card; with a 15–16 GiB cache it ran 12–16 requests concurrently and cleared 26 levels in 30 minutes against 13 for the battle profile and 18 for the same vLLM 0.30 runtime without pruning, and 47 levels in one hour against 37–40 for the old stack in 132 minutes. The Hugging Face listing of the base model held at least six packagings runnable on our server (four NVFP4, two W4A16); the third-placed team turned out to use Intel's W4A16 AutoRound build, which keeps attention, linear attention, shared experts, PLE and MTP in BF16 (99.64% of BF16 on its card's benchmarks). Our weekly scan looked for new models, not for packagings of the model we already served.
- We measured fp8 KV on the wrong question and rejected it. On a nightly build it raised cache capacity by ×1.32 (not the ×2 byte arithmetic suggests), requests by 5–8%, and levels not at all — at an unchanged 32k window. Its purpose in every disclosed solution is different: to make a 70–131k window affordable. rellik13 made the same mistake in mirror image (he tested fp8 KV bundled with a re-quantisation of the non-expert layers, saw a drop, blamed the cache) and reversed it on the last day for +8 points (§12).
- The public bench did not predict the battle. The REAP stack doubled levels on 25 public games at 30–60 minutes per game and scored 4.00 in the competition against 4.30 for the old stack. Kaggle's machine is not the bottleneck (AMD EPYC 9B45, a fast single core, a 600 W GPU). The unmeasured differences are the horizon (up to 2.2 hours per game, four waves) and the hidden games. We therefore only trust a bench for effects measured in multiples; effects of 1–3 points can only be settled by repeated competition submissions.
- Machines differ more than runs. A second rented pod with the same GPU model was ~1.5× slower: vLLM's engine loop pinned one CPU core at 100% while the GPU sat at 37% utilisation and a 450 W limit. Two harness layers measured on it (predict-before-act, observation in the message) could not be compared with the control from the previous pod, and their verdicts stayed unproven. We now record a machine passport (CPU model, single-core Python loop time, GPU power limit) before any run.
- Speculative decoding is assumed, not measured. The model's built-in multi-token-prediction head accepts 2.65–2.95 tokens per pass at three draft tokens; whether that helps or hurts at 14–16 concurrent requests on a CPU-bound engine loop is untested. On the stack of the best disclosed solution (SGLang, a separate INT4 draft, a 64k hot-token draft vocabulary) we measured 2.60 accepted tokens per pass and a median 723 generated tokens/s at ten concurrent requests; one disclosed solution runs with the draft switched off to fit more streams.
- We never lengthened the history. When the pruned model freed 15 GiB for cache at the end of the month, we spent it on concurrency and left the window at 32k. Of all the items in this list, this is the one the disclosed solutions identify as decisive.
- We read the teacher comparison as a model question when it was a serving question. On 26.09 we compared the teacher (Gemini-3.6-Flash via API) with our agent on the same games (§8): at equal move counts it cleared the same number of levels, and its only real advantage was tempo — 2–3 s per move against 57–146 s — while 87% of our request latency was queueing at our own server. We attributed the tempo to the teacher's dedicated API, i.e. to hardware we did not have, and concluded that the lever was answer length. The question we did not ask was what serving stack we could run on the same card. The answer came four days later from outside: the disclosed stack, on our card and our model family, raised our battle score from about 4 to 24–27. The copy changed the server, the model packaging and the harness at once, so the share due to throughput is not separated; that it is most of it is our inference from the game-bench result that score follows total generation (§13) — and throughput is exactly the quantity the teacher had differed in. A distillation effort aimed at the teacher's decisions would have targeted the one thing that was not different.
The general lesson for a fixed-hardware competition is the order of work: model packaging, serving engine and version, cache and what it is spent on (history first, concurrency second), time allocation across games — and only then the harness. We did it in reverse.
12. Milestone 2: the same model at 20–28 against our 4.30
On 30.09 the second milestone closed and four solutions in the top eight were disclosed in full. All run the Duck harness lineage and Qwen3.8-Flash-Next on the same single card. Scores are the public leaderboard; configurations are from the authors' notebooks and write-ups, not from our measurements.
| Franzen | Lord Han Solo | rellik13 | face-of-agi | ours | |
|---|---|---|---|---|---|
| Public score | 27.89 | 23.84 | 22.53 | 20.00 | 4.30 |
| History window (tokens) | 131,072 | 127,488 | 69,632 | 130,816 | 32,768 |
| KV cache dtype | fp8 | fp8 | fp8 | int4 | BF16 |
| KV pool (tokens) | 1,011,264 | 1,417,100 | 1,004,288 | — | ≈105,000 |
| Prefix cache | on | on | on | on | off |
| History trimming | one block, 118k → ~59k | to an 81.5k target | 57k → 45k | append-only | exactly to fit, every request |
| Streams decoding at once | 10 | 14 | 16 | 31 | ≈3 of 16 slots |
| Per-layer embeddings | host RAM | host RAM | host RAM | — | host RAM |
| Model packaging | Intel W4A16 | mixed NVFP4+FP8 | Intel W4A16 | NVFP4, 12.5% experts pruned, GPTQ on ARC data | RadixArk NVFP4 |
| Server | SGLang fork + 5 patches | vLLM nightly + patches | SGLang fork | vLLM + own INT8 kernels | vLLM build of 26.08 |
| Undo action exposed | yes | yes | yes | yes | no |
| Structured world-model sections | off | off | on | on | on |
| Time across games | priority re-queue | fixed cap | shared pool, UCB | token budget per level | fixed cap |
The first place (50.65) and second place (40.97) are undisclosed, and nothing public explains the gap between 28 and 41–51. Everything below concerns the gap between 4 and 28.
The common factor is retained history. Every disclosed solution keeps two to four times more of the game in the prompt than we did, and pays for it the same way: a compressed KV cache, prefix caching, and trimming in large blocks so that the cached prefix survives between trims (Franzen reports 93.43% of prompt tokens served from cache). One author supplies a natural experiment. rellik13's last step, 14.49 → 22.53, changed no prompt, no harness logic, no model and no sampling: the KV cache went from BF16 to fp8 and the freed memory went into history (trims from 37k → 27k to 57k → 45k). His reading of stuck-level transcripts is the mechanism — the agent often had the right goal but lost track of what it had already tried, because trims left a handful of turns on levels that take dozens. His window before that step was the one we ran all month.
The same misreading, independently. rellik13 writes that he kept concurrency high because he took decode throughput for the bottleneck, while his logs showed the KV pool 91–97% full and requests queueing for cache space. Our own false conclusion of 10.09 (§11.1) is the same error. Two teams reading the same symptom the same wrong way suggests the symptom is misleading by construction: a starved cache presents as slow generation.
Added text hurt; removed errors helped. No disclosed solution reports a gain from adding instructions, and two report losses (§2). The text changes that did help were corrections: naming the undo action UNDO instead of the opaque ACTION7, saying "level restarted" instead of "game over" after a death, and deleting a line that told the agent to ignore the bar that is in fact its move budget. Franzen's list of what did not clearly help reads like our §2: structured world-model updates, summarisation of history, composite animation images, stricter verification hints, extra guards, more elaborate priority rules.
Perception that adds information, not description. Franzen names access to intermediate animation frames as the strongest perception change and one game as essentially unsolvable without them. Replaying 9,918 recorded moves of the 25 public games on the local engine, we find that 25% of moves return intermediate frames and 22% contain cells that appear in neither the board before nor the board after; the distribution is bimodal — ten games have such moves at rates of 18–95%, the other fifteen at near zero. The measure depends on which moves the recorded agent made and undercounts mechanics it never reached.
What we checked ourselves. (i) Our copy of the best open notebook runs the regime it describes: fp8 KV, a 139,264-token server context, ten requests decoding at once, 87.2% of prompt tokens from cache over a four-minute run from a cold start, and prompt processing at 2.6% of request time. Four minutes do not reach the first trim, so cache behaviour after trimming is unobserved. (ii) Reasoning effort is not a difference between solutions: the model's chat template defaults to xhigh, and the template is byte-identical across three packagings we compared, including the one we had served all month. (iii) One disclosed technique we had already run: pinning what was learned on a solved level into the prompts of the next. Two one-hour probes at the 32k window showed no signal (5.07 and 4.27 against controls of 6.11 / 5.40 / 3.91); its author reports +2.8 from one pair of single submissions. Neither is evidence beyond run-to-run spread, and the regimes differ.
Our reproduction. We submitted an unmodified copy of the best open solution on 01.10 and it scored 24.25 on the public leaderboard, against 4.30 for our own best build three days earlier. Nothing of ours is in that submission: the whole gap between our September work and the disclosed line is accounted for by the stack beneath the harness and the disclosed harness changes, and none of it by the twenty layers. The same day gives an estimate of single-submission spread that no team could afford to measure alone. Within hours of the disclosures, 77 teams posted a new best score of 20 or more: range 20.67–34.77, mean 26.33, standard deviation 3.09, quartiles 24.22 / 25.90 / 28.40; the original author's 27.89 sits near the middle and our 24.25 at the 26th percentile. We do not know what each team submitted — most are presumably copies of one of the open notebooks, some modified — so 3.1 points is an upper bound on the spread of one build rather than a measurement of it. It is enough to say that a single submission cannot resolve an effect of two or three points on this base, and that the tenth place moved from about 18 to 30.37 in one day without any new idea entering the field. A second submission of the identical copy on 02.10 scored 26.78 (first: 24.25), a difference of 2.5 inside that bound. By 03.10 the 25th place stood at about 31.7, almost entirely copies of the same notebooks.
The same spread on a local bench. Two identical runs of the copy on the 25 public games at a two-hour budget (our bench, which gives each game on average the same server time as the battle: 10 streams × 532 min / ~110 games ≈ 48 minutes per game, divided by the scheduler) scored 46.06 and 48.73 on the harness's per-game score (50.63 and 53.06 on the level-weighted score used in §13 and §13A for the same two runs), with 105 and 107 levels and 7 games fully solved in each. The means agree, but single games flip between runs — m0r0 4.5 → 100, vc33 21 → 100, r11l 100 → 48 — so the difference between a variant and the mean of two controls has a standard error of about 6.3 points from this pair. A single bench run therefore resolves only effects above roughly 12 points; our macro-move layer (a scripted move string executed in one tool call with early stop on level change, death or a no-op) scored 51.61 with 109 levels and 4–9% fewer moves than the controls, +4.2 against their mean — not distinguishable. We submitted it to the battle anyway, as the one component of the current build written by us; it scored 24.85 (03.10), inside the copy cloud.
How the disclosed teams measured. Two practices recur and we adopt both. Runs are compared at an equal number of generated tokens per game rather than at equal wall time; and a local 25-game score is not trusted to rank builds — rellik13 reports a build that scored 37 locally and 9.98 on the leaderboard and another that scored 32 locally and 13.40, and the same notebook submitted three times scored 5.02 / 5.50 / 6.42. This matches §11.5.
13. Serving experiments on the disclosed stack (02–03.10)
Having adopted the disclosed stack (SGLang fork, Intel W4A16, fp8 KV cache, ten streams, 128k window), we audited it the way §11 says we should have audited ours: where does the server actually bind, and what does each memory-saving option cost in model quality. Two instruments made this cheap.
A server bench without games. We record every request the harness sends during a stand (full prompts with images, tool schemas, chat-template arguments) and replay them against a server: each game is a sequential stream, ten or twenty streams at once, and every answer is forced to its recorded length (ignore_eos, max_tokens = recorded completion tokens). The load is then identical for every server configuration, so configurations can be compared on throughput and cache behaviour alone. The bench reproduces itself to about 0.3% (608 vs 610 generated tokens/s for two configurations that should not differ at this load). It cannot measure whether a configuration plays better — only whether it serves the same load faster.
A KV-quality probe. For 20 recorded contexts of 34–70k tokens we compute the mean negative log-likelihood of the model's own recorded answer (first 384 tokens) given the full history and given a truncated 5–8k history. The full-history NLL measures how well the cache preserves what the model needs; the difference between the two ("gain from memory") measures how much the history is used at all. Since the answers were generated with full history, the gain is biased upward and only comparisons between cache formats are meaningful.
What the two instruments found, on a rented card of the competition model and then on Kaggle:
- At the battle configuration the cache does not bind. With ten games and the 128k window the KV pool (1.01 million tokens) peaks at 67% full on the replayed load, which never reaches a trim; on two 132-minute stands (24.10) its maximum was 0.97–0.99, but it stayed above 0.9 only 1.6–2.1% of the time, with no evictions and 93% of prompt tokens come from the prefix cache. A second cache tier in host RAM (HiCache) changes nothing there (610 vs 608 tokens/s) — not because it does not work, but because the recorded histories are trimmed at ~58k and never overflow the pool. Its value for long histories is untested by this bench; we have no recordings longer than 84k.
- A hidden concurrency cap. The model keeps a Mamba state per request and the server reserves five state slots per running request; with the default 60 slots it silently caps concurrency at 12 (
max_running_requests is capped to 12 by the mamba state cache). A configuration we ran as "14 streams" was in fact 12. Any variant with more streams must raise the slot count with it. - Twenty games break the cache, and a host-RAM tier repairs it. Doubling the recorded load to twenty streams (each game replayed twice, the copy tagged so that only the system prompt is shared): throughput rises to ~930 tokens/s while the pool fills (about nine minutes), then the server starts evicting histories and recomputing them — prefix-cache hits fall from 93% to 46%, 187 of 536 requests are served essentially uncached — and over the whole load throughput is 415 tokens/s, −32% against ten games. With a 30 GB host-RAM tier the evicted history is restored instead of recomputed: 86% cache hits, 4 uncached requests, 650 tokens/s, +7% against ten games. Each game then receives half the generation of the ten-game configuration; whether twice the games at half the speed is worth it is a question for the game bench, not for this one.
- fp8 KV costs nothing measurable. fp8 against BF16 on the probe: full-history NLL 0.421 for both formats, difference +0.0026, inside a paired confidence interval of about ±0.01–0.02.
- NVFP4 KV, which did not run on this stack, now does — with our patch. A 4-bit cache would make the pool ×1.5–1.75 larger, but on this server it would not start: FlashInfer refuses a 4-bit cache, and the Triton path crashes on an unknown dtype. The cause is not the attention backend at all. The model's full-attention layers always run through the server's sparse-attention path (QSA), which selects the top-k tokens of the history and copies them into a scratch buffer before calling the attention kernel — and that copy has no notion of packed 4-bit values or their per-block scales. We wrote the missing piece: a Triton kernel that gathers the selected tokens' packed bytes and FP8 block scales and dequantises only those tokens into a BF16 scratch buffer, so the rest of the path is unchanged; the same for the chunked-prefill gather; plus a fix for a CUDA-graph capture failure in the cache write (the scale was passed as a Python float and copied host-to-device during capture), and an option that drops a full-pool FP8 dequantisation workspace the sparse path does not need. Verified bit-exact against the library's own dequantiser on the card, then measured on Kaggle on one machine against fp8: pool 2.11 million against 1.41 million tokens (×1.50); full-history NLL +0.0015 (95% CI −0.005…+0.008), gain from memory +0.0036 (CI −0.007…+0.014); throughput at ten parallel requests 407 vs 403 tokens/s. No measurable loss. At ten games it buys nothing (item 1); it is the cheaper alternative to a host-RAM tier for configurations where the pool binds (item 3), and on the game bench at twenty streams it did not suffice (item 8).
- A newer server version is not automatically better. SGLang 0.5.21 with the disclosed patches ported allocated a pool 18% smaller on Kaggle (823k against 1.01M tokens) and failed to load on the rented card (out of memory while repacking expert weights for Marlin). We keep the disclosed fork.
- On the game bench, the score is the total generation — nothing else we varied moved it. Four configurations against two controls, same 25 public games, one pass each, same kernel length (Kaggle, 03.10):
| | stand | stand 2 | 20 games, fp8 + host-RAM tier | 20 games, NVFP4, no host tier | 250k window, 4 games, host tier | |---|---|---|---|---|---| | Score (harness) | 50.63 | 53.06 | 49.92 | 21.79 | 24.69 | | Levels / moves | 105 / 5 528 | 107 / 5 208 | 104 / 5 037 | 64 / 2 732 | 64 / 2 860 | | Prefix-cache hits | 94% | — | 93% | 38% | 96% | | Concurrent requests | 8–10 | — | 10–16 | 10–19 | 3.3–3.7 | | Generation, tokens/s | 620–700 | — | 650–680 | 850 → 424 | 370–400 | | Levels at equal moves vs stand | — | 89 / 89 | 91 / 90 | 62 / 68 | 61 / 61 |
Twenty games on the NVFP4 cache without a host tier lose to the controls in 22 of 25 games (sign test p = 0.0002): the larger pool (1.27M tokens) still cannot hold twenty histories, the server recomputes them (51M fresh prefill tokens against 11–12M), generation halves and so do the moves. The 250k window keeps its cache intact but leaves room for only four requests at a time, with the same result by a different route. The last row is the control for quality: counting levels only up to the number of moves both runs made on each game, neither the 4-bit cache nor the long history changes what a move is worth (61 vs 61, p = 0.77; 62 vs 68, p = 0.27). Twenty games with the host tier tie the controls: more games divide the same generation, they do not add to it. The tempting summary metric "levels per 100 moves" says the opposite (2.34 for NVFP4 vs 1.90 for the control) and is wrong: early levels are cheap, so a run cut short looks efficient.
- A correctness risk specific to this model. On 02.10 SGLang merged a fix for the host-RAM tier losing the sparse-attention indexer state on eviction; before it, retrieval after a restore from host memory was correct in 0 of 15 cases. The disclosed fork's logs show that it does carry the indexer keys to host memory (
pools=KV + MAMBA + QSA compressed), but the read-side synchronisation that the upstream fix also adds is not visible in logs. Any host-RAM result on this model should be read with that in mind. (Correction, 04.10: on reading the disclosed fork's code, its host-RAM tier saves and restores the n-gram embedding pools before layers are freed; the upstream defect affects SGLang main only [inferred from code]; the indexer's read-side synchronisation was not checked.)
Related harnesses, for scale. Two papers released this week report 100 on the 25 public games with frontier models: VISTA (a visual harness with lossless visual memory the model can query) and Kepler (executable world models validated by retrospective and conditional prediction checks; $778 and 858M tokens for one board). Kepler's paper makes the same point about single runs that §12 does with numbers. We borrowed its stuck-level escalation — a binding directive injected into every turn once a level has absorbed several times its usual cost — and calibrated the trigger on our own controls (median 30 moves per cleared level, so 90 and 180 moves for the two tiers; at 90 it would fire on 6% of levels the agent clears anyway and on 33% of the levels it is stuck on at the end). It is queued, not measured. The executable world model itself does not transfer to our model (§8).
13A. The 25 → 36–39 gap (04–06.10): our own deviations, a fingerprint audit, and a natural experiment
By 05.10 the leaderboard had split into regimes rather than a continuum: Tufa 55.89 and Yi-Chia 48.59; a dense group at 36–39 (the last dance 39.30, mtg 38.33, face-of-agi 37.94, up from 20.00 on 30.09; Lord Han Solo 36.61, up from 23.84); and the copies of the disclosed solution at 25–30. Our battle results on that stack: plain copy 24.25 and 26.78, copy + macro moves 24.85, our build "base3" (copy + macro moves + a faster-decaying game scheduler + a 4× board image instead of 10×) 28.03 and, repeated, 23.62.
Copies are a noise cloud, and ours sits inside it. Other teams' unmodified copies of the disclosed notebook scored 27.89 (its author), 31.47, 27.62, 27.80, 21.01 and 31.73: mean 27.9, SD ≈ 3.9. Our copies are inside that cloud. A "31.54 in a single submission" public fork (a re-fitted scheduler formula) is one draw from the same cloud until repeated; we submitted our copy of it on 06.10 as a one-variable experiment: 27.40, inside the cloud — the scheduler formula is not what separates the regimes.
A fingerprint audit finds no basic difference. Line by line against the disclosed notebook and five public forks scoring 27–32: model artifact and version, draft model, server build, KV dtype and memory fraction, context and trimming thresholds, slots, temperature 0.7, reasoning effort (the template default, xhigh), Docker image — identical byte for byte. The only differences in our battle builds were our own, and two of them reverted choices the author had measured: we replaced his 10× board image with 4× (he names 4×→10× the largest single gain among image settings) and halved his scheduler's decay constants (fitted on 725 level attempts), both on the strength of bench differences of +3.6 and +1.7 — inside a bench noise of ±6.7 — and sent them to battle without a battle pair. A third, a prompt section advising macro moves, is the kind of added advice another team measured as harmful (6.42 → 4.16, 13.40 → 9.98). Lesson: never revert an author's measured choice on a bench that cannot see the difference.
Bench verdicts of the week (25 public games, 2 h, level-weighted score; controls 50.63 / 53.06). Eleven concurrent games instead of ten: 42.81 (95 levels; signs 7:18 against the controls; levels per million generated tokens 19.7 against ~24). An event report printed after every move, meant to save look-only calls: 39.23 — the reports were printed, but moves per call fell (≈1.2 against 1.45) and moves per thousand generated tokens fell from 1.10 to 0.95. Response length: median 1.2k tokens, p90 4.4k; 21–34 responses per run (≈1%) hit the 12,288-token cap and are lost entirely — 6–9% of all generation — and responses above 8k end in a move less often (36–44% against 48%); the disclosed harness already contains a reasoning-effort ladder for truncations, disabled in its battle configuration. Offloading the image hash from the scheduler thread, motivated by a profile that attributed a third of GPU idle time to it, changed nothing on a battle-like load (576.5 / 577.2 against 570.5 / 576.2 tokens/s on a rented card): the profile had been taken with 10× images, the battle build uses 4×. A repetition of the rule of §13: a profile is evidence only for the load it was taken on.
Test-time training, measured honestly. A rank-32 low-rank correction before the output head, trained on the first half of a game's observations, lowers next-token NLL on the second half by 54% (against 44% for a correction trained on another game); on the 30% of observation tokens whose 8-gram never appeared before (not copyable), by 21% against 15%, own better in 20 of 25 games — a game-specific signal exists, mostly shared across games. But the same correction raises NLL on the model's own reasoning and code by 115% (100 steps) to 279% (300 steps), worse than the base in 25 of 25 games: in this form it would make the agent write worse. A KL-retained version is queued; the integration into the server (a per-game adapter bank applied inside captured CUDA graphs, observations collected at prefill, training in a background thread) is written but unmeasured, and we do not expect much of it: the retention that protects the model's writing also limits how much the correction can change its play. (Outcome, §13B: in live play it was harmful.)
A natural experiment: face-of-agi, 20.00 → 37.94. Their public Milestone-2 kernel ("run B", 20.00) is unchanged since 29.09 and identical to the copy we saved. The next build's kernels became private, but its runtime datasets and expert weights, published on 30.09, remain public; comparing them with run B:
- Twice the concurrent games, 31 → 60 (p37) or 76 (p50), paid for in memory: routed experts pruned 37.5% or 50% (against 12.5%; selected by routing mass on their own game runs; the pruned experts receive 4.0% / 9.2% of routings on held-out data), the kept experts re-quantised to 3 bits in groups of 16 (relative weight error ≈15%), the history window cut from 131k to 98k. The KV pool roughly doubles (1,270 → 2,451 blocks).
- A new scheduler: two blocks of ~4.2 h, every one of the 110 games guaranteed its block; a game that has not cleared a level within 90k generated tokens is dropped for good, its slot going to the games with the most levels.
- A plan-execution tool (
run_plan: a move sequence executed in one call, stopping at the first unexpected result) and one prompt paragraph — by their count 52% of responses made no move. - Unchanged: temperature 0.7, reasoning xhigh, 16k answer cap, INT4 KV, the vLLM build, the 4× image, the rest of the prompt and the agent.
Caveat: which build produced 37.94 is not known (45 submissions; the table shows the date of the last, not the best), and best-of-many selection alone is worth a few points at SD ≈ 3. But the mechanism agrees with our one robust finding — the score tracks total generation — and with our own earlier observation on the previous stack (an expert-pruned model, 26 levels against 18 in an hour on a rented card). The trade is explicit: a measurably worse model, served to twice as many games at once.
Porting it. On the disclosed stack routed experts occupy 58.5 GiB of GPU memory and the KV pool 11.5 GiB. The disclosed server builds every MoE layer from one global expert count, so we built a uniform mask from face-of-agi's three nested tiers (12.5% ⊂ 37.5% ⊂ 50%; within a tier the order is unknown): 128 of 512 experts per layer. 25% is the deepest uniform cut the tiers support without pruning the experts they deliberately protect in layers 0–2. It frees ≈14.6 GiB, i.e. a pool about 2.3× larger, room for roughly 20–24 concurrent games instead of 10. Pruning happens at load (pruned experts dropped, kept ones renumbered, router rows sliced; the MTP draft, which face-of-agi does not use and which gives the disclosed stack +37% throughput, is untouched). The probe (06.10): NLL on recorded contexts 0.4209 against 0.4181; greedy continuations diverge after a median of ~10 of 64 tokens (identical in 1 of 20); throughput +1.7% (10 requests) and +2.4% (24) at equal concurrency; on a rented card, pruning with 20 streams gave 663.5 tokens/s against 600.0 for the full model at 10 (+10.6%); battle 22.65 (§13B).
An outside critic's reading, which we adopt. The three regimes are too far apart (25–30, 33–40, 48–56) to be crossed by tuning; the gap is more likely a different configuration of the whole run — model artifact and quantisation, number of concurrent games, time allocation, prompt — than a missing clever layer. Accordingly we froze new harness layers and now run battle submissions in one-variable blocks with a +5 threshold for a "breakthrough".
13B. What changed between 06.10 and 10.10
A transplanted component fails where the whole configuration works. We ported the most visible element of face-of-agi's jump — expert pruning, to free memory for more concurrent games — onto the disclosed stack, keeping everything else. The pruned model lost little on recorded text and the server did generate more, yet the battle score fell below the copy cloud. We then ran the opposite experiment: an unmodified copy of their whole build (pruned and low-bit experts, a low-bit KV cache, many concurrent games, the guaranteed-time block scheduler, multi-move plans, and their own agent), assembled entirely from artifacts they left public. It starts and runs cleanly on the competition card and, at its native concurrency, generates more than twice the tokens per second of our copy of the disclosed solution. It scored 19.52 in battle (11.10), below the copy cloud; on a long public run it cleared 40 levels against 66 for our copy at equal tokens per game (p = 0.04). The lesson is about units of transfer: what produced their jump is a configuration whose parts pay for each other — memory freed by the weights is spent on concurrency, and the scheduler is designed for that concurrency — not a component that can be lifted out.
Two ways to fill the card, and why they do not add up. On one 96 GB card decoding is bound by moving weights from memory to the compute units, and one trip of the weights can serve every request in the batch. At about ten concurrent requests the compute units spend most of a step waiting for weights; a draft model fills that idle compute with guesses that the target model verifies in the same pass, which is why the disclosed stack gains substantially from its multi-token-prediction draft. With several times more concurrent requests the same idle compute is filled by useful work instead, the step becomes compute-bound, and verifying draft tokens only lengthens it. This explains a design choice we had read as an omission — face-of-agi runs without a draft model — and it means the two levers are substitutes rather than complements: a stack is tuned either for few fast streams or for many slower ones. The same arithmetic bounds a common suggestion to switch to a dense model of similar quality: a dense model reads all of its weights for every token and its attention cost grows with history length, whereas the sparse-attention MoE we use reads a small active fraction and attends to a bounded number of selected tokens, so its speed barely depends on how long the agent's history is.
A fine-tuned draft model, and a gain that was predictable before training. We built a trainable replica of the battle draft model (verified against the server's own outputs), extracted its inputs from the server while it re-read recorded agent responses, and fine-tuned it on our own games — first the first draft position only, then the full three-step chain with a training-time attention mask that reproduces how the server's draft attends to its own earlier guesses. Held-out acceptance improved, the training curve flattened early, and the gain on the server was a few percent — far below what would be visible in a battle. The ceiling was computable in advance from the acceptance length already in our server logs: it implied that the first draft position was already accurate and the chain broke at the later positions, and the measured improvement matched the chain formula. The useful outputs are the method (a verified replica, a server-side extraction hook, a chain-training mask) and a rule we failed to follow before spending on it: derive the attainable gain from existing logs before training anything. The limiting factor appears to be data — a few thousand recorded responses from twenty-five public games, used to adjust an already well-trained draft — rather than epochs or the training objective.
Live test-time training is harmful in this setting. The KL-retained correction of §13A, integrated into the server and trained on each game's observations during play, made the agent write longer reasoning before each move and delay its moves; within an hour it cleared several times fewer levels than the controls. Retention that kept the damage small on recorded text did not protect free generation. Where test-time training is known to help — static puzzles with a few examples and a single answer — the model's output is short; here the correction shifts exactly the long free-text generation that pays for moves.
A larger KV cache is stable but buys nothing at ten games. Our 4-bit KV-cache patch (§13.5) ran a long, battle-length phase on all public games with no server error and the pool well filled, and scored within the spread of the fp8 controls: at ten concurrent games the cache is not the binding resource, so the extra room goes unused. The same build returned a zero in battle, for a reason we could not determine — the battle rerun exposes no logs, and the long phase had excluded a server failure. We record this as a risk class rather than a mechanism: paths that exist only in the hidden rerun cannot be exercised by any public run, so a server-level change belongs in battle only when its expected gain justifies that risk.
Score and generation, more precisely. Our data now constrain the relation in both directions: removing generation (fewer tokens per game) lowers the score almost proportionally, while adding it to games that are stuck yields little. Equal time for every game and more concurrent games — the face-of-agi design — therefore beats giving more time to the games that are already progressing.
What is and is not known about the top. The two leading teams have not disclosed their builds. Model artifacts publicly attributed to one of them turned out, on inspection of the kernels that use them, to belong to a different competition; a public snapshot from the other is the model of its first-milestone entry. What can be shown is that teams on the same open MoE model reached the top-ten threshold through system engineering. Whether the gap above that comes from a trained model or from a different harness cannot be decided from public evidence. A participant's public measurement study adds one useful decomposition: almost all of the public-set score comes from the easy two fifths of the games, and on the rest the dominant failure is committing to a hypothesis before probing the new elements of a level. Short launch checks on the easy games — the mode the disclosed notebook uses outside battle, and which we used for launch checks — cannot see this; our comparison benches used all public games.
13C. What the leaderboard measures: thinking with a fixed model versus a model trained for games (14.10)
ARC-AGI-3 is meant to measure how efficiently a system acquires skill in games it has never seen. The leaderboard cannot tell two sources of score apart: a model that reasons better in a new game, and a model that has been trained, through distillation or reinforcement learning on game-like environments, to carry priors about how such games tend to work. Both are within the rules, and the hidden games prevent outright memorisation. They are nonetheless different claims about intelligence.
Our programme stayed, with few exceptions, on one side of this line. The main model's weights were not trained in any battle build before 20.10 (the draft model was fine-tuned from 12.10, and two builds served pruned experts: 22.65 and 19.52); the one battle with a trained adapter, on 20.10, scored 15.73 (§13F). So what we measured is, with these exceptions, the harness and serving around a fixed model. The public record is less clear-cut than it first looked. The best open configuration without weight changes scores in the high twenties when its own author or others resubmit it unchanged; a higher figure shown on its public page turned out to be another participant's submission attributed to it by the platform. One team that started from that notebook reached the top ten by changes it has not disclosed, so we cannot say whether harness work alone can go that far. The two leading teams whose public artefacts we could inspect both train their own models. Within the fixed-model route, every harness change we could separate from noise acted through one quantity, tokens spent per completed level. The single positive single-run result, an instruction to keep private reasoning short (−19% tokens per cleared level, p = 0.035), shortened reasoning uniformly rather than removing any specific kind of restatement; a repeat on other states did not confirm it (98.5k and 99.0k tokens per clear, p = 0.58), and in four battles its build scored 27.23 against 25.27 in two battles for the same copy with the draft model alone (26.61 for the plain copy). Removing reasoning outright, by contrast, collapsed the score.
We read this as a measurement, not an excuse. With a fixed model, the score is the model's in-game reasoning efficiency multiplied by serving throughput, and harness layers move only the first factor, and only by small amounts. Moving beyond that requires changing what the model knows. Whether that knowledge is general skill or a prior over the game distribution is exactly the question the benchmark should answer, and a leaderboard of single scores cannot. We suggest that future rounds report, alongside each score, whether weights were trained, and on what.
Figures for this section — the per-level token comparison, the battle spread of identical copies, and the leaderboard thresholds — are kept in the project log and will be added for the final version.
13D. A paired instrument and its own noise; training on one card (11–16.10)
Battles of the copy and its variants. Unmodified copy of the disclosed solution: 24.25 / 26.78 / 25.45 / 29.96 (mean 26.6). With our chain-fine-tuned MTP draft: 26.44 / 24.09 (25.3). Plus a brevity instruction: 24.36 / 30.04 / 25.51 / 29.00 (27.23). Best single result 30.04 (509th of 4,333 on 15.10; 100th place 33.06, 10th 40.09). A whole-build copy of face-of-agi's p37 configuration: 19.52. A "34.3" shown on the disclosed notebook's page was another participant's submission attributed to it by the platform; its author's own score is 27.89.
The level-start twin test. Two kernels differing in one setting play the same 16 control levels (levels the model usually clears), 5 attempts each from the level's start state, 80k-token cap per attempt; metric: generated tokens per cleared level and clears, with a per-level sign test; ~2 GPU-hours per kernel. Single-run results against the control (61/80, 59.6k):
- no brevity instruction: 54/80, 73.6k (brevity −19%; 12 better / 3 worse levels, p = 0.035; a repeat on other states on 17.10 did not confirm it: 98.5k and 99.0k tokens per clear, 43 and 44 clears, 8:5, p = 0.58); reasoning shortened uniformly (~6% of characters per call, −13% tokens per call) — the share of board restatement and hand-counted paths unchanged;
- board image 4× instead of 10×: 55/80, 74.3k (+25%), 12 worse / 2 better, p = 0.013; +12% calls; calls printing the board as text 42% against 35% (ls20 0.40 → 0.76, sb26 0.32 → 0.63); 16×: 56/80, 67.0k, n.s.;
- rule-check tools (encode/predict checked by the harness against every recorded transition of the level): 0 calls of 556 when only mentioned in the system prompt; with a stall reminder, 114 calls, 58 exposing a wrong rule, 26 code errors; repaired to full agreement in about a third; 59/80, 64.0k; a Kaggle repeat 47/64, 63.8k — no gain in two runs;
- p37 weights under our harness: 55/80, 72.1k (+21%, 5/9, p = 0.42); their harness spends 78k tokens per cleared level against 31k for ours;
- answer cap 6k with effort step-down: 58/80, 63.3k, n.s.; stall-context reset: 7/81 clears against 13/81 with history; time reallocation between games, replayed: −3…+0.6 points.
The instrument's own noise. The control build run twice: Kaggle 61/80 at 59.6k, a rented card of the same type 53/80 at 77.7k (+30%; levels 4:10, p = 0.18). Run-to-run spread is 15–30%, the order of every single-run effect above. A per-level sign test does not absorb a global shift between runs, so single-run p-values overstate certainty; we report the effects above as signals requiring repetition. (A pod-versus-Kaggle machine difference is not excluded.)
Training on one card. LoRA (r16, q/k/v/o — only the 12 full-attention layers, 5.7M parameters) on the Intel W4A16 model in transformers 5.19 + auto-round + peft, the 95 GiB n-gram embedding table on the CPU through hooks. Initially 45 tokens/s; a stacked-experts path (one dequantisation per layer, grouped matmul) brought 55–67. A per-module profile of a 4,096-token forward pass (31.6 s): sparse-attention indexer 21.2 s, Gated DeltaNet 7.3 s, experts 2.7 s, n-gram table 0.02 s. The indexer's reference implementation loops in Python over every query token. Our vectorised indexer (block keys computed once, scores and top-k as matrix operations over query chunks) matches the reference mask exactly (0 differences on 3,000 tokens of a real layer) and is 68× faster per layer; training reaches 719 / 781 / 807 tokens/s at 8.2k / 10.3k / 12.3k tokens (peak 86.8–92.4 GiB; longer examples do not fit). Computing logits only over the target span saves ~11 GiB at 12k tokens. An epoch over 7.0M tokens of curated data is ~2.5 GPU-hours. Serving: the disclosed SGLang server loads a LoRA on attention; a zero adapter reproduces outputs at −5% single-stream speed. The adapters trained in this setup were worse than the base model — for a reason that was not the method (§13E).
13E. Training the model that is served: two silent loading errors, a likelihood check, and the first reward-selected adapter (16–19.10)
Symptom. Every adapter trained on one card between 16 and 18.10 was worse than the base model on held-out data: a world-model adapter wrote verified simulators for 0 of 80 held-out tasks against 8 of 80 for the base; a moves adapter (STaR on hinted successes) cleared 0 of 24 level starts against 1 of 24. We first read this as the method failing on a small model; both verdicts were obtained on the broken training path described below and are invalid.
Instrument. We compared per-token log-likelihoods of the same 3,000-token prompts between the training path (transformers) and the serving stack (SGLang), with and without a random adapter. The server's own spread sets the scale: the identical request sent twice differs by a median of 0.06–0.10 nats per token (mean ≈ 0.4; a fifth of tokens > 0.5, plausibly near-tied block selection in sparse attention). Speculative decoding, the draft's token map and prefix caching do not affect input log-probabilities (verified in the server source).
Two errors, both silent. (1) The checkpoint stores the 95 GiB n-gram embedding table as 128 shards; transformers 5.19 does not recognise them and leaves the table randomly initialised, reporting it only as a MISSING line in the load report. We had handled this trap in a custom loader in September and lost it when switching to the stock loader. (2) Our fast mixture-of-experts path dequantised without the GPTQ zero-point offset (zeros are stored minus one), shifting every expert weight by one quantisation step (mean error 0.014 against mean |w| 0.040); its own unit test compared against our own packing in the same wrong format and passed. With the table fixed but not the experts, training-versus-server divergence was still a median 0.81 nats; with both fixed it fell to 0.07–0.10 — the server's own noise. A strong random adapter (r64 on attention and Gated DeltaNet) is then served exactly: error with the adapter equals the noise, the served effect equals the reference (median 0.42 against 0.39), each part (attention, DeltaNet) separately, and doubling the adapter doubles the effect. The lesson generalises: a fast replacement must be checked against someone else's reference, and a trained model against the served one, before any training result is interpreted.
Reward-selected self-distillation. Data: turns of the base model's own runs that were needed for a cleared level (labelled by replaying each run on a deterministic engine replica and greedily removing actions), shorter than their pool median, capped per level (5 for easy games, 15 for medium and hard), plus hinted successes with the hint removed; held out entirely: five public games. An adapter (r16, attention only) trained on 1,005 such turns, 61% of an epoch: held-out loss 0.802 → 0.661; first-move agreement with the successful move on 171 held-out turns 25.2% → 35.7% (26 against 8 discordant pairs, sign test p = 0.003); replies 486 → 416 tokens (median). Completing the epoch: agreement 38.6%, replies 351 tokens. On nine held-out level starts × 4 attempts through the battle harness: 7/36 clears against 3/36 for the base (Fisher p = 0.31), 369k against 858k generated tokens per clear at equal total generation, and no loss of serving throughput (624 against 598 tokens/s). A second round — the adapter playing 75 training-game level starts without hints — cleared 25 of 150 attempts and yielded 155 new turns. The adapter scored 15.73 in battle (20.10, §13F); the second-round adapter was not measured in full play.
Open. The harness shows the model the board as text and as an image; training sees text only (images are stripped from every example). The adapter still played better with images present, but whether training with images helps is unmeasured.
13F. Where the model fails on levels it sometimes solves, and training at battle format (20–21.10)
A reward-selected adapter can cost more than it gains. The first adapter trained on the model's own successful moves (§13E) scored 15.73 in battle against 24.4–30.0 (mean 27.2) for the same build without it. A one-hour stand on the 25 public games ruled out a serving fault: tokens were generated at the same rate, but at 2.2 M tokens the adapter had cleared 54 levels against 80, and the gap widened with depth [measured]. The training set had passed three filters (answers no longer than the median, moves that survived a hindsight shortening of the winning path, inputs stripped of images and truncated to 12k tokens), and the held-out examples had passed the same filters. The −17.5 % held-out loss (pilot, 61% of an epoch) and the 23 → 39 % move agreement (full epoch, the adapter used in battle) therefore rewarded the filter's style — shorter and more decisive — rather than better decisions [inferred]; the adapter made 60 moves per level against 36 already on the first levels.
There is room that self-training could target. Over eight long stands of one build family, scoring each game by its best run raises the mean from 45.7 (one run) to 67.5 (best of eight), and from 17.5 to 42.0 on the 13 hard and medium games [measured]. About 22 points sit on levels the same model clears in another run; the efficiency of cleared levels accounts for only 2.5–3.3.
Restarting a stalled level with a fresh context does not collect that room. A simulation on the same stands, with per-level clear costs estimated by Kaplan–Meier, gives +0.5 points at the best threshold and ≤ 0 if a restart succeeds 30 % less often than a fresh run [inferred]. Attempts on levels that other runs clear end by budget at a median of 1.55× the median clear cost; long loops that a restart would cut are rare.
The model usually finds the goal; it fails on mechanics. For 20 levels that the base model clears in some runs and not others, two successful and two failed attempts each were checked against the game source. The correct goal appears in 40/40 successful attempts (median: third call) and in 36/40 failed ones (median: second call). The failures divide into mechanics 26, time 9, execution 3 and goal 2 [measured, judged against source]. A recurring pattern is a correct goal that the agent later abandons as "unreachable" because a mechanic is not understood (a hidden part of the board, a block that passes through a barrier, a ghost that replays a recorded path). This motivated two changes: a harness note that keeps the stated goal in the last message and asks the agent to test untried interactions before abandoning it, and engine-verified "what happens if" questions built from real states of recorded runs (about 2 000 per pass, each answered exactly by the local engine).
Cross-training at battle format runs end to end; in its second round it hurt play. Two adapters were trained on Flash-Next with the battle input (history and board images), each on the successful levels of the other half of 26 recorded base runs, and then played eight held-out official games for 15 minutes each on the same machine as an untrained control: 11 and 13 levels against 13 [measured] — indistinguishable, with no sign of the harm above. The training itself was small (0.43 M and 0.28 M tokens), because about half of the examples did not fit in memory without the fast linear-attention kernel; with that kernel training ran 2.3× faster. The second round (21.10; 60 minutes on the same eight held-out games, one machine, one run per arm): the base cleared 25 levels and the two adapters 13 and 6, although held-out loss fell 14.1% and 13.0%; the share of generation spent after the last move was 2.8% for the base against 30.9% and 66.9%; examples longer than ~9.5k tokens had been dropped silently, leaving 134 and 111 [measured]. Training for battle was closed on 22.10.
13G. On-policy RL in miniature: a laptop and one Kaggle GPU (26–28.10)
The loop runs end to end; its first gain was a board-blind rule. To prepare group-relative RL on the model's own play (the method behind the top of the leaderboard, as far as the public tooling suggests), we built it at small scale on a laptop. The local engine replays recorded wins to starts 1–6 moves before a level is cleared (132 starts verified by replay, 44 levels from 23 games). Each start gets a group of 8 attempts with the current weights; the reward is 1 if the level is cleared within 2k+2 moves; the advantage is the reward minus the group mean; there is one LoRA step per batch, so training stays strictly on-policy. A 1.7B model answers with one move, its first token restricted to legal moves. The first night run raised the win rate on training starts from 7 % to 25 % [measured], but on every start the trained policy matched a fixed rule — "press 5, otherwise 4" — that wins exactly those starts without looking at the board [measured]. The baseline for a learned policy here is therefore the best board-blind rule, chosen on training starts and scored on held-out ones, not the untrained model.
With that baseline and a KL penalty, eleven steps show nothing. The second run added the level's move history to the input, a KL penalty toward the base model, and a split of levels into training and held-out sets. Two evaluations before training gave 2.5 % and 6.2 % on training starts and 16.7 % and 10.4 % on held-out starts; after 11 steps, 6.2 % and 20.8 % [measured; 4 attempts per start]. The board-blind rule chosen on training starts scores 20.0 % there and 16.7 % on held-out starts. Per start, the changes go 3:1 in both sets (sign test p ≈ 0.6), within the spread between the two pre-training evaluations [measured]. The binding constraint was signal: in 5 of the 11 steps no group had mixed rewards, and in the other 6 only one group of four did [measured]. A 1.7B model wins too rarely from these starts to produce a gradient. This is the "10–90 %" condition the method states; before any larger run, the first measurement is the base model's win rate on the starts, not the training curve. [Status: not carried through — 11 steps, one seed.]
A 20B model meets the condition; reasoning effort does not change it on one-move starts. Without training, gpt-oss-20b (3.6B active parameters, answering directly in its final channel) wins 13.8 % of attempts on the same training starts and 27.1 % on held-out ones, and its outcomes differ within the group on 13 of 32 starts against 6 for the 1.7B model [measured; 4 attempts per start]. We then let it reason first at each of its three effort settings, with an 8,192-token cap after which the answer is forced. On the 26 starts all four settings completed, the win rates were 18.3 % (no reasoning), 13.5 % (low), 15.4 % (medium) and 26.9 % (high); on the 12 starts one move from a cleared level, 35.4 / 25.0 / 33.3 / 33.3 %, and per start no setting differs from no reasoning (high vs none 4:5, sign test p = 1.0) [measured; 4, 2, 1 and 1 attempts per start]. The cost does differ: about 250 reasoning tokens per call at low, 4,800 at medium, and at high the cap on every call, spent largely on re-reading the board row by row [measured]. The one hint runs the other way on starts three moves out: high cleared 3 of 14 against 2 of 56 attempts without reasoning (Fisher p = 0.05 by attempt, 3:1 by start, p = 0.62), and not on the starts a board-blind rule clears [measured]. On the pre-registered rule (reasoning is trained only if it beats no reasoning one move out at p < 0.1), the first full run trains the no-reasoning policy, and only on one-move starts, where a group of 8 has mixed rewards about three times as often at under half the calls per attempt [inferred from the 4-attempt measurement]. [Status: measurement complete for the decision; the k=3 hint not carried through — one attempt per start.]
At 20B on a Kaggle GPU, the policy learns its training games, not the task. We moved the loop to a single Kaggle GPU: an SGLang server generates with the current LoRA loaded on the fly, a separate bf16 copy takes the gradient step (r = 8 on attention, group of 8, KL 0.05, importance-ratio mask), and the model reasons at low effort before each move. Levels were split by whole games: 82 one-move starts from 16 games for training, 50 from 9 games never seen in training for evaluation; the decision rule, written before the run, asked for at least +5 points on the unseen games at p < 0.05 by level, a shift a power simulation detects 84 % of the time. In 63 steps (8.3 h) the training-set win rate rose from 23.9 % to 29.3 % (+5.3 points from the unrounded rates) (levels 29:18, permutation p = 0.04) and the unseen-game rate from 21.7 % to 24.0 % (levels 24:12, p = 0.16; games 4:4, p = 0.34; 90 % bootstrap interval over games −0.8 to +5.3 points) [measured; two evaluations on each side, 4 attempts per training start and 8 per held-out start]. Almost all of the training gain sits on the 21 starts a board-blind "press 5, otherwise 4" rule clears (+17.3 points against +1.2 on the other 61) [measured]. Loading the adapter on the laptop and sampling one reasoned first move per start shows what changed: on training starts the trained policy chooses 5 or 4 on the blind-rule starts twice as often (29 % → 57 %, and move 6 overall falls from 65 % to 48 %), while on the unseen games it does not (50 % → 12 % on their 8 blind-rule starts), and the probability it assigns to the correct move without reasoning is unchanged in both sets (0.48 → 0.48) [measured; one sample per start]. The adapter learned which moves pay in the games it trained on [inferred]. [Status: transfer to unseen games not seen and not carried through — 63 steps, 16 training games, one seed. What a fair test needs is many more games, and training starts on which a game's most frequent winning move does not win.]
Removing the shortcut without removing starts. Filtering out the starts a board-blind rule clears would leave 37 training starts, only 13 of them ever cleared before training, almost all by clicks — too little signal. Instead every training group draws one permutation of the non-click action numbers; the level history in the prompt is rewritten with the permuted numbers, and the model's answer is mapped back before it reaches the engine. A number then carries no fixed meaning within a game, so "press 5 in this game" stops paying and the effect of an action can only be read from the history; evaluation stays unpermuted. On the laptop the mapping round-trips on 24 of 24 checks (the permuted correct answer clears the level). The rerun starts from the base model with the same split and the same pre-registered rule (unseen games ≥ +5 points at p < 0.05 by level and by game), plus a second condition: the training gain must not sit on the blind-rule starts.
Result (29.10; 64 steps, 7.5 h; runs/mac_rl/kaggle_rl_low_perm/). Figure f20 shows all four runs side by side. Training starts 25.2% → 30.8% (+5.6 points; levels 32:11, p = 0.01; games 11:1). The gain no longer sits on the blind-rule starts: +7.1 points there against +5.1 on the 61 others (g9: +17.3 against +1.2). The nine unseen games: 22.6% → 27.3% (+4.6 points; levels 20:9, p = 0.02; games 7:1, p = 0.02; 90% bootstrap interval over games [+2.2, +7.6]). Against the g9 adapter, final to final, unseen games +3.3 points (games 7:0, p = 0.016). Calibration: the four evaluations of the base model and the two final evaluations of the same adapter differ at p > 0.2 by the same test. By the letter of the pre-registered rule the size is 0.4 points short of +5 while both significance conditions hold; one seed. A reference for the headroom. A rule that reads only the history, "repeat the level's last move", clears 58% of the unseen starts (29 of 50) and 45% of the training starts (37 of 82; replayed on the local engine), against 27% for the trained policy [measured]. This says as much about the starts as about the policy: one move before a recorded clear, the clearing move often repeats the previous one [inferred]. The policy's gain is not concentrated on the starts this rule clears (+5.6 there, +4.3 elsewhere). Removing the shortcut moved learning from action preferences to the starts that need the history, and part of it transferred to unseen games; using the level history as well as this rule does is the next target, together with starts further from the clear, where repetition alone no longer suffices. A first-move probe on the laptop (one sample per start, no power for a 5-point effect) shows no change on the first move of unseen games (probability of the correct move without reasoning 0.479 → 0.482) and no return of the shortcut (share of 5/4 on rule starts 0.57 → 0.48, against 0.29 → 0.57 for g9) [measured]; the evaluation gain, if it holds, would then sit in moves 2–4, after the policy has seen the effect of its own first move [not tested].
13H. Fifteen battles of one configuration, and an audit of the harness itself (28–30.10)
The battle spread. By 01.11 we had submitted four builds of the disclosed Milestone 2 configuration seventeen times: the plain copy (24.25 / 26.78 / 25.45 / 29.96), the copy with our fine-tuned draft (26.44 / 24.09), with a brevity instruction (24.36 / 30.04 / 25.51 / 29.00) and with a keep-alive pulse that stops an idle-time rule from ending games while the server loads (24.09 / 26.67 / 30.85 / 22.12 / 21.58 / 22.92 / 30.97) [measured]. Together: mean 26.18, SD 3.04, range 21.58–30.97; the four build means (25.3–27.2) do not differ (one-way ANOVA p = 0.83) [measured]. The pulse build scored 30.85 on 28.10, then 22.12, 21.58, 22.92 and 30.97 on the next four days [measured]. A fifth build adding a goal note and a server watchdog scored 19.73 / 26.57 / 25.58 / 24.16, mean 24.01 (Welch p = 0.26 against the seventeen) [measured]. Our best score, 30.97, is the upper edge of the spread (z ≈ 1.8 against the other sixteen) and 21.58 its lower edge (z ≈ −1.7), from the same build [inferred: not an improvement and not a regression]. Over the nineteen battles to 30.10 there is no trend (Spearman ρ = −0.23, p = 0.35); the two lowest family scores came on 29–30.10 (p ≈ 0.01 for that pattern, a test chosen after seeing the data), and the same build then scored 22.92 and 30.97 [measured]. The same 181–185 GB model spent 4 to 74 minutes reading its weights on our development runs (server log), and loads started at 00–01 UTC were slow in 7 of 8 cases [measured; observational]. A public forum thread reports 10–60-minute starts and slow-starting machines running about 10% slower [forum, not verified]; on our two slow-loading development runs generation was not slower (777 and 783 against ~775 tokens/s) [measured]. Without logs of the scored runs we cannot tell a streak of slow machines from a platform-wide decline [inferred]. Each submission's private score comes from the other half of the same run, so a run that drew good conditions helps both halves [inferred from the published split].
The audit. After ten weeks of layers on top of the harness we audited the harness itself: the exact battle code (the disclosed bundle, the author's patch and our brevity patch, ~15k lines) and the request logs of one 25-game, 132-minute development run of the same build (1,941 calls, 4,667 moves, 97 levels). Three agents read context, loop and scheduling independently; the main findings were checked by hand.
- Move accounting is exact (4,618 moves + 24 automatic resets + 25 warm-up resets = 4,667) [measured]. No defect worth more than one or two levels on 25 games was found.
- Past reasoning is 42% of the model's input (45% above 80k tokens) and its only written memory: the model wrote 820 tokens of visible text in the whole run [measured]. When history overflows (69 times; 115k → 56k tokens, ~55 messages at once) the harness drops whole turns; 34 of the 69 drops cut into the level being played, leaving a median 6 of its 13 moves [measured]. Dropping reasoning first is not the remedy — removing it from history cost 2.91 against 9.43 (§2) — and images, 13.9% of input with 17.8% of them byte-identical copies, are the untested candidate [measured shares; remedy not tested].
- 52% of calls make no move (902 only inspect, 93 end in a Python error) [measured].
- A false "no change" after animations survives (34 of 240 no-op marks, clustered on stalled levels); a fix measured on 11.10 was indistinguishable on a long stand [measured].
- After ten consecutive failed requests a game is abandoned for good; a server watchdog that restarts the server cannot bring it back [measured from code].
- Weight loading varies fifteen-fold. The same 185 GB loaded in 300 s on 03.10 and in 3,977 / 4,412 s on 24.10, and development runs start their game clocks before the server is ready: one control lost 41 of 132 minutes [measured]. Development runs from different days are therefore not comparable (part of the 46–49 against 40–42 between 03.10 and 24.10) [inferred]. A timeline simulation of 110 games from recorded runs puts 30 → 75 minutes of loading at about −3 battle points, a candidate explanation of part of the battle spread [assumed; simulation accuracy ±1–2 points].
- The chat template asks for maximum reasoning effort while each turn asks for brevity; a lower effort setting measured on 12.10 was indistinguishable (8/82 against 12/82, p = 0.58) [measured].
We made no change to the battle build from this audit: the deadline left no room for the stand runs that behaviour-changing fixes need, and the zero-risk switches (not abandoning games, not re-sending truncated reasoning) are each worth well under the battle spread [inferred].
14. Reproducibility
Every number above comes from a recorded run or a script in the project repository, and the counterfactual analyses (score budget, truncations, search bounds, goal induction) run offline on a deterministic replica of the engine, so they can be re-executed without a GPU. The battle notebook is sergueimakarov/arc3-graft-dfranzen-m2-cbattle (version 3, https://www.kaggle.com/code/sergueimakarov/arc3-graft-dfranzen-m2-cbattle?scriptVersionId=360443322; its seven battles scored 21.58–30.97; later versions add the goal note). The early Kaggle kernels behind the live measurements (not public) are arc3-stock-flash (baseline), arc3-stock-flash-predhint (goal-predicate layer), arc3-lora-canary-j (runtime adapter) and arc3-merge-canary (weight overlay).
15. What we contribute
- A corrected conclusion, with the evidence against our own first reading. We attributed twenty null layers to the model's ceiling. The disclosed Milestone 2 solutions show that the same model, harness lineage and card support 20–28 points against our 4.30 once the agent keeps its own history, and we document where in our month each signal pointing there was available and misread (§2, §11, §12).
- A negative result with a narrower mechanism than we first claimed: in a serving regime with a 32k window, twenty harness layers that add text, structure or search did not move the score; the missing ability is not dynamics; and the open reading is that such layers displace the history the agent needed.
- A measurement discipline: baseline spread from ≥3 identical runs; pre-registered thresholds; counterfactual evaluation on a deterministic replica, which predicted a layer's sign before compute was spent; and controls that re-fit parameters to a pseudo-goal, overturning two of our own headline numbers.
- A goal vocabulary with measured per-kind transfer, turning "the agent lacks goals" into numbers a successor can build on.
- A score budget that tells a successor which levers can matter before any of them is built, and two negative counterfactuals (best-of-two short passes, time reallocation) derived from recorded runs at zero compute cost.
- Where the executable-world-model recipe stops working: on a small open model it fails at writing an exact simulator, not at planning — which argues for moving observation work into the harness rather than asking the model for a simulator.
- The engineering of changing the core inside a sealed kernel: the exact serving constraints a runtime adapter imposes, and a weight-overlay route that imposes none — with the byte-level patching rule that makes a pinned-hash checkpoint modifiable without re-uploading 135 GB.
- NVFP4 KV cache for the model's sparse attention (§13.5): a patch that makes a 4-bit KV cache work on the disclosed SGLang stack, where it previously could not start — dequantising only the selected tokens inside the sparse-attention path, plus a CUDA-graph capture fix — with a ×1.5 larger pool and no measurable loss in a likelihood probe.
- A server bench and a cache-quality probe (§13): replaying recorded harness load at fixed answer lengths compares server configurations to ~0.3%, and a full- vs truncated-history likelihood probe compares cache formats; together they located where the cache starts to bind (above ~12 games at a 128k window), a hidden concurrency cap (five Mamba slots per request), and what repairs the overflow (a host-RAM tier: −32% → +7%).
- Macro moves for the agent (a move string executed in one call with early stops), the component of our current battle build that is our own; on the bench it is not yet distinguishable from the control.
- A fingerprint audit and a natural experiment (§13A): the copies of the disclosed solution form a noise cloud (SD ≈ 3.9) that contains ours; our only deviations were reversals of the author's measured choices; and a byte-level comparison of a team's public builds before and after a +18 jump points to expert pruning traded for twice the concurrent games — which we port to the disclosed server as a uniform 25% cut that keeps the MTP draft.
- Units of transfer and two ways to fill the card (§13B): a transplanted component of a better configuration failed where a copy of the whole configuration runs at more than twice our throughput; and an account, in terms of weight traffic and idle compute, of why many concurrent streams and speculative decoding substitute for each other on a fixed card.
- A draft-model fine-tuning pipeline and its predictable ceiling (§13B): a verified trainable replica of the battle draft, server-side input extraction, chain training with a training-time mask, and the observation that the attainable gain followed from acceptance statistics we already had.
- Verifying that the trained model is the served one (§13E): a per-token likelihood comparison calibrated by the server's own spread, which exposed two silent loading errors that had made every adapter worse than the base, and the first positive training result after the fix — reward-selected self-distillation on the model's own successful moves.
- What fails on levels a model solves only sometimes (§13F): a best-of-k decomposition that locates ~22 points on levels the same model clears in other runs; a restart simulation that shows why a fresh context does not collect them; and a source-checked measurement that the model finds the goal early (40/40 and 36/40) but abandons it when a mechanic is not understood — with a filter-induced failure of self-training as the cautionary case.
- RL on game starts learns per-game action shortcuts (§13G): at 20B on one GPU the training gain sat on starts a board-blind rule clears and did not transfer to unseen games; permuting action numbers per group removes the shortcut without removing starts, after which unseen games gained +4.6 points (p = 0.02, one seed) — with room to grow: on these starts repeating the level's last move clears 58%, against 27% for the policy.
- The battle spread and a harness audit (§13H): fifteen battles of four builds of one configuration (SD 2.85; build means indistinguishable, ANOVA p = 0.71); where the harness loses the current level's history, why games are abandoned when the server fails, and why development runs from different days are not comparable.
The natural extension to histories (ever / never / count / before / within / action-run) covers all 53 levels and fails its control completely: a random trajectory prefix, where no level was cleared, is "separated" in 986 of 986 trials. Coverage without a control is worthless.
What would change our mind: a progress measure that separates winning from sibling actions at AUC ≥ 0.6 using only a transferred goal, or evidence that the next-level rate rises on ≥60 first-level games. Absent either, our reading is that on a fixed small-for-the-task model the binding constraint is turning a known goal into a plan, not naming the goal.
What did change our mind. The paragraph above was written before Milestone 2 and names evidence inside our own frame. The evidence that arrived was outside it: a solution that changed only memory and gained eight points. The goal and progress measurements of §§4–6 remain valid as measurements, but their role as an explanation of the score is unproven until they are repeated in the long-history regime, where the agent may name and pursue goals it previously lost between trims.
Limitations. Every harness verdict in §§2–8 is conditional on the 32k serving regime and none has been repeated on the disclosed stack. The server bench of §13 measures throughput under a fixed load, not play; the cache-quality probe uses answers generated with full history and compares formats only; the NVFP4 result is one probe on 20 contexts and one throughput check, not a game bench. Configurations and score steps of other teams are as reported by their authors; single-submission differences of a few points, theirs and ours, are within the spread we measured for identical builds. The gap to the 36–39 group is attributed in §13A to more concurrent games bought with expert pruning, from one team's public artifacts; that attribution is unverified, and the gap to 48–56 is unexplained. All live measurements are on the 25 public games; the hidden set is roughly four times larger and our public-to-hidden transfer ratio was 0.28–0.53 across three September builds (0.36 for the Flash-Next base), so absolute numbers do not carry over even though the counterfactual comparisons do. The score budget is computed on three base runs and inherits their spread. The delivery experiments prove that modified weights load and take effect; they do not prove that a trained adapter helps, which remains the open question and the one we would spend money on next — first one epoch on the cheapest surface, then a measurement, and only then the rest. Finally, our own attempt at that surface in an earlier experiment under-fitted badly (0 levels of 16 with the experts frozen), which is the honest prior against the idea we are recommending others evaluate.
Appendix A. Everything we tried, with how it was measured and what came of it (August – 27 October 2026)
Not a claim of novelty: several items have relatives elsewhere (per-state action probing, per-game manuals, executable world models). The point is the record — each item was built, run and measured. [m] measured, [i] inferred.
| Area | What we did | How we measured | Outcome |
|---|---|---|---|
| Base model | gpt-oss-120b, Qwen3.8-27B, Flash-Next (several quantisations), Swift 1.5, REAP-448 | battle / stand / bench | Flash-Next is the strongest that fits 96 GB; gpt-oss-120b failed on harmony history in Duck (0/25 untrained, one run — not a verdict on it as a base for training) and did not start under the disclosed SGLang build (21.10); gpt-oss-20b ran on SGLang 0.5.19 on the same Kaggle card for RL (27–29.10) [m] |
| Serving | vLLM → SGLang; KV-cache size, fp8/int4/NVFP4 KV, HiCache, 10 vs 20 concurrent games, MTP draft (trained chain), FR-Spec, 4/5-step speculation | server bench replaying recorded load; stands | server was the binding constraint in September; 10 games fp8 stays; 20 games + HiCache equal; speculation >3 steps does not start [m] |
| Harness layers (~20) | notes, goal gates, macros, structured reasoning request, concise reasoning, history summary, park/early-cut schedules, image ×4/×10 | paired stands, ≥3-run noise floor | none distinguishable from noise except the server fix; history summary and early cut harmful in single runs [m] |
| Copying the disclosed Milestone-2 solution | graft of Franzen's notebook and face-of-agi; fingerprint audit of 40+ public copies | battles | copy 24–30 (our best 30.97 on 01.11, a build of the same family); copies form a noise cloud, SD 3.1 (77 teams' new bests on 01.10, an upper bound) to 3.9 (six known copies) [m] |
| Run safety | warm-up pulse (RESET every 8 min until the server answers) after two zero scores | fake-server harness test; battle | one zero since — 22.10, a build that had the pulse; none in the 8 battles since 23.10; score unchanged (v3 against v1, Welch p = 0.35) [m] |
| Scoring analysis | RHAE decomposition; best-of-k over 8 stands; fresh-context restart simulation | recorded runs | efficiency costs 2.5–3.3 points, unsolved levels ~50; best-of-8 +22; restart simulation +0.5 [m/i] |
| Failure analysis | 33 stalls checked against game source (10.10); goal probe: 20 levels × (2 wins + 2 losses) (21.10) | agents judging against source code | goal found in 40/40 wins and 36/40 losses; failures: mechanics 26, time 9, execution 3, goal 2; correct goals abandoned when a mechanic is not understood [m] |
| World models | algorithmic ensemble; CNN; model-written simulators (CEGIS); Memento-style code-as-model agent; training "write a simulator" | exact replay of recorded transitions; games | ensemble ≤30 %; CNN median 0 %; exact simulators in ~25 % of games; code-as-model agent 2/63 levels twice; training on teacher pairs harmful (1/192 vs 9/192) [m] |
| Training (LoRA) | built a training path matching the served model (two silent loading bugs found); text and battle-format (images, history) inputs; reward-selected moves; cross-training of two adapters over two rounds; starts on sometimes-solved levels | held-out loss; move agreement; battles; same-machine game checks vs base | offline metrics improved every time (−13…−17 % loss); in play three adapters were worse (battle 15.73 vs 27.2; round 2: 13 and 6 levels vs 25) and two indistinguishable (11 and 13 vs 13 in 15 min); on nine held-out level starts the battle adapter cleared 7/36 vs 3/36 (p = 0.31) [m] |
| Test-time training | in-game adapter updates | stands | harmful: reasoning ×2.3 longer [m] |
| On-policy RL in miniature (§13G) | group-relative RL with LoRA from starts 1–3 moves before a cleared level, on a laptop and one Kaggle GPU; 1.7B and gpt-oss-20b; four reasoning settings | unseen games against a board-blind rule chosen on training starts | 1.7B: first gain was the blind rule, then no shift (no signal); 20B: +5.3 on training games, +2.2 on unseen ones (p = 0.16) — it learned which moves pay in its training games; with action numbers permuted per group, +4.6 on unseen games (p = 0.02 by level and by game, one seed), reference: repeating the level's last move clears 58% of these starts against 27% for the policy [m] |
| Teacher data | frontier-model play and reasoning as targets | — | Claude agents asked to write reasoning for training targets were stopped by the provider's filter (20.10) and were used only as judges for measurement; Gemini teacher data trained the August students |
| Mechanics data | engine-verified "what happens if" questions from real states (~2 000) | base accuracy (not run; dropped 21.10) | a rule that ignores the model answers 64.6% of the questions; unusable as a measure as built |
| Harness for keeping a thought | goal note with a "don't abandon" guard; tried-actions summary; cross-level rule ledger; death ledger; automatic action probing at level start; rule board shared across concurrent games; per-turn counter log | 25-game stands on one machine | goal + tried summary: 69 levels vs ~77 for a reference stand at equal tokens (no gain) [m]; full set in progress |
| Infrastructure | local deterministic engine replica (561/561); one-process training on rented GPUs; Kaggle queue tooling; fake-server harness tests | — | every harness change verified on the real harness before a paid run |