How models learn
← The series

Chapter 03

The slope, and the gradient

Which way is down, for one parameter and then for all of them.

Nudge one parameter, watch how much the loss moves, and divide. Shrink the nudge until the answer settles: that is the slope. One of those per parameter, stacked into a column, is the gradient.

  • nudge
  • derivative
  • gradient

The image it is built onFog on a hillside. You cannot see the bottom, you can feel the ground tilt under your feet, and that is enough to take one step.

The loss has to come down, and the only things you can change are the parameters. So the question is which way to move each one. It turns out to be measurable rather than guessable, and the thing you measure has a name.

The question this chapter answers

You have a number that says how wrong the machine is, and a pile of parameters you are allowed to change. Change one and the loss moves. Change it the other way and it moves the other way.

So: for each parameter, which direction makes the loss smaller, and how much does it care? Both answers come from the same measurement, and the rest of the chapter is that measurement.

Nudge one parameter and watch

Hold every parameter still but one, and plot the loss against that one. Move it by 0.1 and the loss falls from 4.00 to 3.60: 0.1 of parameter bought 0.40 of loss, a rate of 4.

Use a nudge of 2 instead and the rate comes out different, because the curve bends between the two points. So shrink the nudge. Try 1, then 0.1, then 0.01, and the rate moves less and less until it stops moving in the digits you are watching.

That settled number is the slope at that point. It is what you get by zooming in until the bend disappears, and it is the whole measurement.

Drag the nudge towards zero and watch the rate stop changing. Then move the point along the curve and do it again somewhere steeper.

The plot takes focus. Left and right arrow keys move the measurement point along the curve, Home and End take it to either end. The sliders and the readout below carry the same values.

losswhΔ losssecanttangent
w
1.800
h
2.00000
loss(w)
1.868608
loss(w + h)
1.306688
(Δ loss) / h
-0.280960
The line through the two points swings as the nudge shrinks, until it lies along the curve and the rate settles.

The sign says which way, the size says how far

Negative slope, go right. Positive slope, go left. And the size warns you what to expect: a large slope means a small change moves the loss a lot, a small one means you are on a plateau and this will take a while.

it falls, so go rightit rises, so go left
The same curve twice. The tangent and the arrow beneath it share a colour, so the decision is read rather than computed.

One slope per parameter, stacked

Moving two parameters at once tells you nothing, because you cannot say which one did what. So hold the second still and measure the first, which is exactly what you have just done, then swap them over. Two parameters, two numbers. A thousand parameters, a thousand numbers.

That list is what the word gradient names, and nothing more mysterious than that. Written out, each line asks the same question about a different parameter: if I nudge this one, how much does the loss move?

With two parameters the pair of numbers can be drawn as an arrow on the plane of settings, and that arrow points at the fastest way up. You want down, so you go the opposite way. That is the whole of the minus sign in the rule the next chapter is built on.

Sweep the arrow all the way round and watch the rate it reads. One direction wins, and it is not asserted: it is found.

PARAMETER APARAMETER BTHE BOTTOM-2.590

Sweep it round, then drag the point and sweep again · Ring = no change, outside it the loss falls · Tick = best tried · The trace is scaled to fit the dial; the rate itself is in the readout

Nothing tried yet · sweep the arrow round

Point
-1.20, 0.90
Loss
6.401
Angle
0.0°
Rate
-2.590
Best rate
—
Best angle
—
The plane of the two parameters. Every direction is tried, and the trace records how fast the loss falls along each one.

The slope

The settled rate has a name: the derivative. Here h is the nudge you have been shrinking, and w is the parameter you are nudging.

rate=loss(w+h)−loss(w)h\text{rate} = \frac{\text{loss}(w + h) - \text{loss}(w)}{h}

Once h is small enough for the answer to settle, that answer is written ∂L/∂w. It is one thing, the amount the loss moves when the parameter moves, and not one number divided by another.

The gradient

One slope per parameter, stacked into a column. That column is the gradient, written ∇L:

∇L=(∂L∂w1∂L∂w2⋮∂L∂wn)\nabla L = \begin{pmatrix} \dfrac{\partial L}{\partial w_1} \\[0.6em] \dfrac{\partial L}{\partial w_2} \\[0.4em] \vdots \\[0.2em] \dfrac{\partial L}{\partial w_n} \end{pmatrix}

Read it as a direction, and nothing more. The next chapter turns it into a move by going the opposite way to it, by an amount you choose.

Nudge a parameter, shrink the nudge until the rate settles, and you have a slope. One per parameter, stacked, is the gradient: a direction, measured rather than guessed.

A direction is not a distance. How far to go before measuring again is the next chapter.

The misreading to head off

The gradient does not point at the minimum. It points at the steepest climb right here, which in a curved valley sends you into the opposite wall rather than along the floor.