Chapter 03
The slope, and the gradient
Which way is down, for one knob and then for all of them.
Nudge one knob, watch how much the loss moves, and divide. Shrink the nudge until the answer settles: that is the slope. One of those per knob, stacked into a list, is the gradient.
- nudge
- derivative
- gradient
The image it is built onFog on a hillside. You cannot see the bottom, you can feel the ground tilt under your feet, and that is enough to take one step.
One knob, one curve
Fix everything but a single knob and plot the loss against it. The first half of this chapter happens on this one curve.
A nudge, and an exchange rate
Move the knob by 0.1 and the loss falls from 4.00 to 3.60. So 0.1 of knob bought 0.40 of loss, which is a rate of 4. Use a nudge of 2 instead and the rate comes out different, because the curve bends between the two points.
Shrink until it settles
Try a nudge of 1, then 0.1, then 0.01. The rate moves less and less, until it stops moving in the digits you are watching. That settled number is the slope at that point, and it is what you get by zooming in until the bend disappears.
The plot takes focus. Left and right arrow keys move the measurement point along the curve, Home and End take it to either end. The sliders and the readout below carry the same values.
- w
- 1.800
- h
- 2.00000
- loss(w)
- 1.868608
- loss(w + h)
- 1.306688
- (Δ loss) / h
- -0.280960
Drag the nudge towards zero. The line stops swinging and lies along the curve, and the rate stops changing. The sign decides, the size warns
Negative slope, go right. Positive slope, go left. A large slope means a small turn changes a lot; a small one means you are on a plateau and this will take a while.
The same curve twice. The tangent and the arrow beneath it share a colour, so the decision is read rather than computed. Now do the other knob
Turning two knobs at once tells you nothing, because you cannot say which one did what. So hold the second still and move only the first, which is the curve you have just been reading, then swap them over. Two knobs, two numbers. A thousand knobs, a thousand numbers. That list is what the word gradient names, and nothing more mysterious than that.
Why the pair points somewhere
With two knobs the pair of numbers can be drawn as an arrow on the plane of settings. Spin the dial through every direction, read the rate you get, and the best one turns out to be that arrow.
Drag the point or the arrow · Ring = no change, outside it the loss falls · Tick = best tried
Nothing tried yet · sweep the arrow round
- Point
- -1.20, 0.90
- Loss
- 6.401
- Angle
- 0.0°
- Rate
- -2.590
- Best rate
- —
- Best angle
- —
The steepest direction is not asserted, it is found. Every direction is tried, and one of them wins. Which is why the step has a minus
The arrow points at the fastest way up. You want down, so you go the other way. That is the whole of the minus sign in the rule the next chapter is built on.
That settled rate has a name: the derivative. It arrives last, for something you have been handling all chapter. Written out, with h standing for the nudge you have been shrinking:
Show it in symbols
rate = ( loss(w + h) − loss(w) ) ÷ h
Once h is small enough for the answer to settle, that answer gets written dL/dw. It is one thing, the amount the loss moves when the knob moves, and not one number divided by another.
One slope per knob, written as a list. Every line asks the same question about a different knob: if I nudge this one, how much does the loss move?
Show it in symbols
gradient = [ slope for knob 1, slope for knob 2, … ]
The list is usually abbreviated to ∇L, and the step written w ← w − (step size × ∇L). That is the sentence above in symbols: go the opposite way to the list, by an amount you choose.
The misreading to head off
The gradient does not point at the minimum. It points at the steepest climb right here, which in a curved valley sends you into the opposite wall rather than along the floor.