When the whole team is AI
Notes from a project lead: what three months of managing AI agents taught me
ARC Prize 2026 · 27 October 2026 · written before the competition's code deadline, so without scores or configurations
For the last three months my engineering team has been made of AI agents. I am a project leader, not a machine-learning engineer. I entered ARC Prize 2026, a Kaggle competition on interactive reasoning puzzles, and ran it the way I would run any project: a goal, a budget, a backlog, reviews. The difference was the team.
The lead engineer was Claude Code, an AI agent working in my editor, on my laptop, on rented cloud GPUs and on Kaggle's weekly GPU quota. It wrote the code, ran the experiments, and handed narrower jobs to sub-agents of its own: searching the web, reading other teams' code, reviewing its own conclusions. Outside critics were other models, Gemini and ChatGPT, which I consulted in their own sessions with prompts built from the project documents. None of them had a stake in the work, and that turned out to matter.
This is not a post about whether AI agents can write code. They can, quickly and well. It is about what it takes to manage them when real money, real deadlines and real decisions are involved. Every rule below exists because something went wrong first.
What the lead agent is good at
It does not get tired, it does not get bored, and it reads the source code of a library faster than I can find the documentation. It ran hundreds of experiments over the season, kept a detailed log of each, and rebuilt infrastructure that would have taken a small team days. Tonight, while I was asking questions, it moved a training pipeline onto a separate inference server, tested the whole thing on my laptop against an imitation of that server, and launched two runs. That is the upside, and it is real.
The downside is that an agent has no sense of cost, no memory beyond what you write down, and a strong pull towards whatever task is directly in front of it. Each of those showed up as a concrete failure.
Seven ways it went wrong, and the rule each one produced
1. It treated agreement as permission. On 14 September I wrote "let's do all of it" about a list that mixed free work with paid runs. The agent launched two runs worth about nine hours of compute; I had them taken down within the hour. The same thing happened on 19 October, when my choice between two options was read as an order to launch, and on 24 October, when "we could raise the temperature, of course" was read the same way.
Rule: every spend of money or quota needs one line from the agent, "Launching <run>, about N hours?", and an explicit yes to that exact line. Agreeing with an idea is not permission to spend on it.
2. It was too eager to clean up. On 2 October an automatic watchdog the agent had set up deleted a rented GPU machine a minute after an experiment failed. The fix was found five minutes later. The machine, and the chance to test the fix on it, were gone.
Rule: nothing that costs money is stopped, deleted or rented without my direct command at that moment. No automatic killers.
3. It paid for the same thing many times. Loading a large model onto a rented GPU takes about ten minutes. On 24 September the agent ran each probe as a fresh process: six crashes, four full reloads, about $105 burned. On 16 October the pattern returned: three reloads, and after two hours training had still not started.
Rule: an expensive resource is loaded once, in one long-lived process, and all further work is fed into that process.
4. It stopped watching. On 4 October a run died nine minutes after launch because of a missing package. Nobody noticed for two and a half hours: the agent's background watcher had died together with its session, and the machine kept billing.
Rule: a watchdog runs outside the agent's session from the first minute, and the first ten to fifteen minutes of every run are checked by hand, because that is when most failures happen.
5. It reached verdicts too early. "Harmful", "doesn't work", "closed": the agent liked these words, and in early September its errors clustered exactly there. One such verdict burned a run; others reversed completely once someone counted. One idea was declared harmful after a single one-hour run in a setup that could not have shown anything either way.
Rule: every claim in a report is labelled measured, inferred or assumed. "Refuted" is allowed only when the check is as strong as the one behind the alternative. Otherwise the status is "not carried through", and the item stays open.
6. It ran experiments that could not change anything. On 17 October a five-hour run finished with a result that would not have changed a single decision whichever way it went. On 27 September a day of paid runs produced no verdict at all, because the expected effect was smaller than the noise.
Rule: before a paid run, write down "if X, we do A; if Y, we do B". If A and B are the same, the run is not needed. Before any spend there is also a short card: the expected effect, the noise and the threshold, and what has already been checked for free.
7. It forgot. Long agent sessions get compressed, and requirements fall out. When artifact names changed, the project backlog silently stopped being updated for two weeks. Findings that said "we should do X" stayed as sentences in a log. A timestamp written from memory instead of the clock produced a false alarm that a run had been going for five hours.
Rule: a living backlog that the agent must read at the start of every session (a hook puts its summary in front of the agent automatically), updated in the same turn as every launch. Every finding becomes a task or a written "we won't, because". Times come only from the clock.
Over the season, by my own accounting, skipped checks cost about $150–200 in rented compute. That is small money, but every dollar of it was avoidable, and none of these rules existed on day one.
Why one agent is not enough
The lead agent optimises inside the frame it has already chosen. It rarely questions the frame itself. An outside critic does. In August a critique from a second model, brought back as a short specification, produced in about two hours what a week of the lead agent's careful tuning had not, at roughly a tenth of the cost. The critic was wrong on several details and right on the points that carried the weight.
Rule: run important conclusions past a second model that has no stake in them. Implement the cheap parts of its critique at once and measure, rather than arguing first. When it lands a point, re-check the lead agent's "later" pile, because more of it is usually cheap than the lead agent thinks.
The failure that rules did not fix
Rules made the agent safe and efficient. They did not give it direction.
For most of the season the agent did what it does best: it took the strongest open foundation available, built by other teams, and added careful, well-measured improvements on top. Each one was tested honestly. None moved the result beyond the noise of the foundation itself. Meanwhile the teams at the top of the leaderboard, as far as one can tell from outside, were doing something different: training their own models, and several of them building on a core idea they had carried over from previous years' competitions.
Tonight I asked whether we had contributed anything of our own to the score. The honest answer, which the lead agent confirmed: useful measurements, a small speed-up, but no method. The question came from me, not from the agents. Agents will dig wherever you point them, tirelessly and well. They will rarely tell you that you are digging in the wrong place, and even the critics answered the questions I asked rather than the one I should have asked.
That part of the job is not delegable. The bet, meaning what we believe and what we are trying to build that nobody else has, has to come from the person.
How I would start the next project
- Start with the bet, not the code: what is our own idea, and which assets must exist by when.
- Measure the noise before comparing anything.
- Write the rules above down on day one, not after the losses.
- Treat the lead agent as a brilliant, tireless engineer with no sense of cost and no memory, and give it both in writing.
- Give every important conclusion a second, independent model as a critic.
- Keep money, launches and direction with a human.
The full write-up of the competition, with every number, will be published here after the code deadline on 2 November.