Chapter 09
The training loop
Everything, end to end.
Predict, measure, assign the blame, adjust, and start again. Four moves that fit in ten lines of Python, one number to set, and a curve whose shape is worth learning to read.
- passes
- training curve
- diagnosis
The image it is built onA day in the workshop. Produce, inspect, adjust the machines, start again tomorrow.
Every piece is now built: a prediction, a number that says how wrong it is, a slope, a step, a handful of examples at a time, a network, and the rule that carries blame back through it. What is left is the loop that puts them in order and runs them again.
Four moves, on one figure
Predict, measure, hand out the blame, adjust. Each station on the cycle links back to the chapter that built it.
Press One station four times, then press Run.
four sizes of that one step
- at
- predict
- passes
- 0
- steps
- 0
- loss
- 12.365
- step size
- 0.0833
A step of 0.0833 is the only number here you get to pick. Four presses of One station is one adjustment; press Run to let it go on its own.
Watch one run
Start a small network and let the loss curve fall in real time. This is the first thing anyone does, and it is worth doing slowly once.
The one number left to choose
Look at what the loop above actually needs from you. The forward pass is settled, and has been since chapter 6. The backward pass is settled, and was the whole of chapter 8. The parameters are whatever they happen to be, and the data is the data.
What is left is how far to move at each step, and that is one number. Every preset on the figure is that number and nothing else: too timid is a small step, too bold is a large one, and between them the run either settles, or wobbles its way down, or throws itself off the map. Nothing else about the run changes when you press them.
Press Too timid, then Too bold, and watch the step size in the readout. It is the only thing that moved.
Three shapes worth knowing
A curve that flattens too early, a curve that oscillates, a curve that climbs. Each has a cause, and each is available as a preset so the shape can be produced on purpose.
One pass through the data
The batches of chapter 5 are six examples out of twenty-four, so four adjustments is exactly one pass through the whole training set. The counter in the middle of the ring counts those passes. The word people use for one of them is epoch, and you will meet it everywhere; it is worth recognising and it is not worth using here, because what it means is what it is called above.
Ten lines of it
Nothing in the four moves needs a library. Here is the entire loop in plain Python, and every line of it has had a chapter to itself.
for pass_number in range(200): for batch in batches(train_data):ch. 5 guess = forward(model, batch.x)ch. 6 error = loss(guess, batch.y)ch. 2 blame = backward(model, error)ch. 8 for p in model.parameters: p -= step_size * blame[p]ch. 3, 4That is training, whole. Four moves in a ring, one number to set, and a curve whose shape tells you which of the four is going wrong.
Everything so far has run on numbers that arrived as numbers. The next chapter is what happens when what you have is words.
The misreading to head off
A falling loss is not proof of anything good. It only says the model is fitting what it was shown, which is what the tenth of the data kept back in chapter 5 is there to check.