Learning curves
What neural networks taught me about learning
Parallels between training machines and learning from life — observations, not science
I spend a lot of time watching neural networks learn. After a while their problems start to look oddly familiar: how big a step to take, what counts as progress, when a plateau is just a plateau, what to remember and what to let go of.
A caveat before anything else: an artificial neural network is not a brain. The name is a loose borrowing from biology, and the resemblance ends quickly. These parallels are not about how the brain works — they are about learning itself. Anything that improves through experience runs into the same handful of problems; machine learning simply had to name them precisely.
Taking steps
1. Big leaps overshoot
In machinesEvery update nudges a network in the direction that reduces its error. How far it moves each time — the learning rate — is one of the most consequential settings in the whole process. Too large, and training jumps over the valley it was aiming for, or blows up entirely. Too small, and it crawls. A common recipe: start gently, take bigger steps once things are stable, and make them smaller again towards the end.
In peopleChanging everything on the first of January is a learning rate of ten. It feels decisive and tends to end in a spectacular divergence by February. Small corrections, made often, get further — and it is fine to make them finer as you get closer to where you want to be.
2. Explore while the road is long
In machinesAn agent that always picks the best option it knows will never find a better one; an agent that only explores never benefits from what it has found. Reinforcement learning has to balance the two, and the usual answer is to explore more early on and exploit more as time runs out — a discovery is worth most when there is plenty of time left to use it.
In peopleA good argument for curiosity at the start of anything — a new job, a new city, a new field: ask around, try the odd approach, meet people you would not normally meet. Later it makes sense to be more selective. Being choosy in the first week and adventurous in the last is exactly the wrong way round.
3. Most dead ends are not dead ends
In machinesWhen training stalls, it is tempting to think the network is stuck in a pit. In very high-dimensional spaces, though, a genuinely bad pit is rarer than it looks; many flat stretches are saddles — level in most directions, still sloping down in a few. Optimisers cope using momentum: they keep part of their previous speed, which carries them across the flat part instead of stopping there.
In peoplePlateaus feel like walls from the inside. Usually they are not: there is still a direction that goes down, it is just not the obvious one, and the slope is gentle. A steady routine works like momentum — it keeps you moving through the flat weeks when no single day seems to matter. Momentum has a downside too: it can carry you straight past where you meant to stop.
Learning from experience
4. You learn most where you sometimes fail
In machinesWhen a model learns by trial and error, one popular approach is to let it attempt the same task several times and compare the attempts with each other. A task it always solves teaches it nothing: every attempt looks equally good. A task it never solves teaches nothing either. The signal lives in the middle, where some attempts work and some do not — so it pays to pick problems at exactly that level.
In peoplePractising what you already do well feels productive and teaches very little. Throwing yourself at something hopelessly beyond you feels heroic and teaches about as much. The useful zone is the slightly uncomfortable middle, where you win sometimes, lose sometimes, and the outcome can still surprise you.
5. Success stories teach the look, not the move
In machinesShowing a model only successful examples works up to a point: it learns what success looks like. What it does not learn is what made the difference — and winning and losing attempts are often almost identical, apart from one or two decisions. Methods that compare good attempts with bad ones put the weight exactly on those decisions.
In peopleBiographies of successful people are full of habits that plenty of unsuccessful people also had: early mornings, cold showers, unshakeable self-belief. The useful question is not “what did the winners do?” but “what did they do differently from those who tried the same thing and did not make it?” The second group rarely writes books.
6. Learn the doubts, not just the answers
In machinesA small model can be trained to imitate a large one. It learns noticeably better from the teacher's full spread of probabilities — “probably a cat, possibly a fox, certainly not a truck” — than from the final answers alone. The teacher's hesitations carry information about how the world is structured. Geoffrey Hinton called this dark knowledge.
In peopleThe most valuable thing a mentor can share is often not the decision but the options they nearly chose and why they did not. “I went with A, B was close, and C was never on the table” teaches far more than “do A”. It is worth asking about the hesitations.
Measuring progress
7. You get exactly what you measure
In machinesA model does not optimise what you meant; it optimises what you wrote down. Reinforcement learning keeps a long list of agents that found loopholes — the best known is a boat-racing agent that scored more points by circling and hitting the same targets over and over than by ever finishing the race. Nothing was broken. The measure simply was not the goal.
In peopleMeasure hours at the desk and you will get hours at the desk; measure meetings and the calendar will fill up beautifully. Neither is quite the same as results. Every so often it is worth checking whether the number you are improving still means what it meant when you chose it. Economists call this Goodhart's law; most organisations call it quarterly targets.
8. One run is not a verdict
In machinesTrain the same network twice from different random starting points and the results can differ noticeably; the error also jumps around from one batch to the next. Experienced practitioners learn not to celebrate or panic over a single run. They repeat the experiment, look at the smoothed trend, and first measure how big a difference has to be before it means anything.
In peopleOne brilliant day does not prove a new habit works, and one terrible day does not prove it doesn't. Weight, mood and sales figures all wobble on their own. Before drawing a conclusion, it helps to ask whether the change is bigger than the usual wobble — and to give it a few more tries.
9. The reward arrives late
In machinesIn a long game the win or the loss comes only at the end, and the learner has to work out which of hundreds of earlier moves deserve the credit. This is the credit assignment problem, one of the oldest and hardest in reinforcement learning. The decisive move is rarely the last one.
In peopleWe tend to thank or blame whatever happened just before the outcome — the last meeting, the last diet, the last piece of advice. Often the real cause was months earlier and far less dramatic: a course taken a year ago, a relationship looked after, a habit nobody noticed. Results arrive with a long delay; memory works with a short one.
Knowing what you know
10. Memorising is not understanding
In machinesGive a network enough capacity and it can simply memorise its training examples, noise and quirks included: perfect on those, poor on anything new. That is overfitting. The remedy starts with honesty — keep aside examples the network never trains on, and judge it only by those.
In peoplePeople overfit too. One bad experience with a dentist, a bank or a business partner can harden into a rule about all dentists, banks and partners. Practising the same twenty exam questions produces excellent results on those twenty questions. The test that matters is the situation you have not seen before.
11. Fluency is not knowledge
In machinesA language model can write a perfectly fluent, confident paragraph about a book that does not exist. It is trained to produce plausible text, and plausible is not the same as true; on top of that, many tests reward a confident guess over an honest “I don't know”. Good systems are judged on whether their confidence matches their accuracy, not on how sure they sound.
In peopleThe most convincing person in the room is not necessarily the one who knows. That includes the voice in your own head, which can be wonderfully articulate about things it has never checked. “I don't know yet” is an underrated sentence — in people and in machines.
12. What leaves the window is gone
In machinesA language model only sees a limited window of recent text. Whatever has scrolled out of it no longer exists for the model, however important it was. Systems that have to work for hours keep notes outside the window and read them back when needed.
In peopleOur window is not much bigger. Last month's insight, last year's lesson, the reason behind a decision — gone, unless it was written down. A notebook is not a sign of a poor memory; it is a sign of knowing how memory works.
The analogy has its limits. A network does not get tired, does not get bored, and never has to explain its loss curve to its family. But anything that learns has to take steps, judge its progress and decide what to keep — and in that sense we are all in the same business.