Chapter 04
Gradient descent
A model is a point. Learning is a walk downhill.
A model is a set of numbers. This one has two, so it can be drawn twice: as a line through the data, and as a single dot on a landscape of error. Everything that follows comes from those being one object rather than two.
Data space
Drag the middle of the line to raise it or either end to tilt it, and watch the dot.
- observations
- the line
- residuals
- lowest point
Parameter space
Drag the dot across the surface and watch the line redraw itself.
- the line
- path taken
- lowest point
- L
- 13.94
- w
- −0.600
- b
- 2.300
- ∂L/∂w
- −11.78
- ∂L/∂b
- 3.784
- steps
- 0
how far each step moves the dot
how far the points scatter from the true line
The line is the dot
The line’s tilt and height are the dot’s two coordinates, w and b. There is one object here and two ways of drawing it. Every line you could possibly draw sits somewhere on that surface.
Drag the dot into a far corner, then put the line back by hand.
Every mistake, added up
Each vertical segment is one mistake: the gap between what the line predicts and what was actually observed. Square them, average them, and one number stands for all of them: the loss. That number is the height of the loss surface, which is why the dot has somewhere to fall.
Make the segments as short as you can by hand, and note the loss you reach.
Which way is downhill
Stand anywhere on the loss surface and one direction leads down more steeply than any other. It can be calculated straight from the data, which is why learning never has to try every line and compare them. The two numbers in the readout, ∂L/∂w and ∂L/∂b, are that direction: the gradient.
Drag the dot somewhere hopeless, then press Step once and watch which way it goes.
How far each step goes
The gradient gives a direction, not a distance. The learning rate, η, is how far you commit to that direction before stopping to look again. Too small and the dot spends a hundred steps on a journey worth ten; too large and it keeps landing past where it meant to stop.
Set η as low as it goes, press Run, and see how long the dot takes to arrive.
Steps that leave the map
Past a certain step size the descent stops working at all. The dot overshoots the valley and lands further up the far side than it began, so the next correction is larger still.
Drag η to its maximum and press Run.
A valley with two floors
The surface above has a single lowest point, so every path downhill arrives at it. Most surfaces do not. Here the same rule meets a valley with two floors. Step downhill, as far as η allows, and where the ball starts decides which floor it settles on.
- Step
- 0
- x
- n/a
- Loss
- n/a
- Slope
- n/a
- State
- descending
| The line | ŷ = wx + b | the whole model is two numbers, one for tilt and one for height |
|---|---|---|
| The loss | L = (1/n) Σ (ŷᵢ − yᵢ)² | squaring makes every miss count as a positive amount, and the average turns n mistakes into one score |
| Sensitivity to tilt | ∂L/∂w = (2/n) Σ (ŷᵢ − yᵢ)xᵢ | a point far out along x has more leverage on the tilt, so its miss counts for more |
| Sensitivity to height | ∂L/∂b = (2/n) Σ (ŷᵢ − yᵢ) | raising the line helps exactly as much as the misses lean one way rather than the other |
| The step | w ← w − η·∂L/∂w, b ← b − η·∂L/∂b | subtract, because the gradient points uphill, and η decides how much of it you take |
A model is a set of numbers. The loss scores them, the gradient says which way to move, and η says how far to go before scoring them again. That loop is the whole of learning; the rest is a question of scale.
Two knobs can be set by looking at the total error directly; millions of them, stacked in layers, need a way for the blame to travel backwards to each one, which is backpropagation.