Chapter 09
The training loop
Everything, end to end.
Predict, measure, assign the blame, adjust, and start again. Five moves that fit in ten lines of Python, and a curve whose shape is worth learning to read.
- epochs
- training curve
- diagnosis
The image it is built onA day in the workshop. Produce, inspect, adjust the machines, start again tomorrow.
Four moves, on one figure
Predict, measure, hand out the blame, adjust. Each station on the cycle links back to the chapter that built it.
this problem stops converging above 0.2082- at
- predict
- epoch
- 0
- steps
- 0
- loss
- 12.365
- step size
- 0.0833
Four presses of One station is one adjustment. Press Run to let it go on its own.
The whole of training. One station is one adjustment, and the curve beside the ring is the run those adjustments produce. The three presets are three shapes worth knowing. Watch one run
Start a small network and let the loss curve fall in real time. This is the first thing anyone does, and it is worth doing slowly once.
Three shapes worth knowing
A curve that flattens too early, a curve that oscillates, a curve that climbs. Each has a cause, and each is available as a preset so the shape can be produced on purpose.
What an epoch counts
One epoch is one pass through the whole training set. The word is defined by what it counts, not by translating it.
Ten lines of it
Nothing in the four moves needs a library. Here is the entire loop in plain Python, and every line of it has had a chapter to itself.
for epoch in range(200): for batch in batches(train_data):ch. 5 guess = forward(model, batch.x)ch. 6 error = loss(guess, batch.y)ch. 2 blame = backward(model, error)ch. 8 for knob in model.knobs: knob -= step_size * blame[knob]ch. 3, 4The smallest machine learning there is. Each line carries the number of the chapter that built it.
The misreading to head off
A falling loss is not proof of anything good. It only says the model is fitting what it was shown, which is what the tenth of the data kept back in chapter 5 is there to check.