Chapter 04

Gradient descent

A model is a point. Learning is a walk downhill.

Chapter 3 measured which way to move. This chapter takes the step, over and over, and watches where it ends up. The model here has two parameters, which is few enough to draw twice at once: as a line through the data, and as a single dot on a landscape of error.

Data space

Drag the middle of the line to raise it or either end to tilt it, and watch the dot.

observations
the line
residuals
lowest point

Parameter space

Drag the dot across the surface and watch the line redraw itself.

the line
path taken
lowest point

L: 13.94
w: −0.600
b: 2.300
∂L/∂w: −11.78
∂L/∂b: 3.784
steps: 0

Learning rate η

how far each step moves the dot

Noise

how far the points scatter from the true line

The line is the dot

The two parameters are the line’s tilt and its height, w and b, and they are also the dot’s two coordinates. One object, drawn twice: move either picture and the other moves with it. Every line you could possibly draw sits somewhere on that surface.

Drag the dot into a far corner, then put the line back by hand.

Every mistake, added up

Each vertical segment is one mistake: the distance between what the line predicts and what was actually observed. Square them and average them, which is chapter 2, and one number stands for all of them. That number is the height of the surface on the right, and it is the only reason the dot has somewhere to fall.

Make the segments as short as you can by hand, and note the loss you reach.

Which way is downhill

This is the gradient of chapter 3, on two parameters: one slope each, ∂L/∂w and ∂L/∂b, both in the readout. Together they point the way the loss climbs fastest, so the step goes the other way. And they are worked out from the data rather than searched for, which is why learning never has to draw every line and compare them.

Drag the dot somewhere hopeless, then press Step once and watch which way it goes.

How far each step goes

A direction is not a distance. The learning rate, η, is how far you commit to that direction before stopping to measure again. Too small and the dot spends a hundred steps on a journey worth ten; too large and it keeps landing past where it meant to stop.

Set η as low as it goes, press Run, and see how long the dot takes to arrive.

Steps that leave the map

Past a certain step size the descent stops working at all. The dot overshoots the valley and lands further up the far side than it began, so the next correction is larger still, and the one after that larger again. It leaves the map in a handful of steps.

Drag η to its maximum and press Run.

A valley with two floors

The surface above has a single lowest point, so every path downhill arrives at it. Most surfaces do not. Here the same rule meets a valley with two floors, and where the ball starts decides which floor it settles on. Press Run, then Reset for a different start, and watch the same rule reach a different answer.

Learning rate0.080

this curve stops coming down above 0.116

Step: 0
x: n/a
Loss: n/a
Slope: n/a
State: descending

Press Run, or take one step at a time. The line through the ball is the slope where it stands.

The line	$\hat{y} = wx + b$	the whole model is two numbers, one for tilt and one for height
The loss	$L = \frac{1}{n} \sum_i \left( \hat{y}_i - y_i \right)^2$	squaring makes every miss count as a positive amount, and the average turns n mistakes into one score
Sensitivity to tilt	$\frac{\partial L}{\partial w} = \frac{2}{n} \sum_i \left( \hat{y}_i - y_i \right) x_i$	a point far out along x has more leverage on the tilt, so its miss counts for more
Sensitivity to height	$\frac{\partial L}{\partial b} = \frac{2}{n} \sum_i \left( \hat{y}_i - y_i \right)$	raising the line helps exactly as much as the misses lean one way rather than the other
The step	$w \leftarrow w - \eta \, \frac{\partial L}{\partial w}, \qquad b \leftarrow b - \eta \, \frac{\partial L}{\partial b}$	subtract, because the gradient points uphill, and η decides how much of it you take

A model is a set of numbers. The loss scores them, the gradient says which way to move, and η says how far to go before scoring them again. That loop is the whole of learning; the rest is a question of scale.

Two parameters can be set by looking at the total error directly; millions of them, stacked in layers, need a way for the blame to travel backwards to each one, which is backpropagation.