Chapter 05 · Not written yet
The gradient
Every slope at once.
Freeze every knob but one and you are back to a single slope. Do it for each knob in turn and the answers stack into a column. That column is the gradient.
- partial slopes
- direction
- steepest ascent
How this chapter will go
Treat the gradient as a list of slopes, one per knob, and say why the step goes the opposite way.
The image it is built onA valley seen from above, until there are more than three knobs. Then the valley is dropped on purpose and the list stays.
Two knobs at once
The cost now depends on two settings, and turning both at the same time tells you nothing: you cannot say which one did what.
So freeze one
Hold the second knob still and move only the first. That is exactly the previous chapter, on a curve, and it gives one number. Repeat with the other knob and you have two.
The list is the gradient
Two knobs, two numbers. A thousand knobs, a thousand numbers. The word gradient names that list, and nothing more mysterious than that.
Why the pair points somewhere
With two knobs the pair can be drawn as an arrow on the plane of settings. Spin the dial through every direction, read the rate you get, and the best one turns out to be that arrow.
Drag the point or the arrow · Ring = no change, outside it the cost falls · Tick = best tried
Nothing tried yet · sweep the arrow round
- Point
- -1.20, 0.90
- Cost
- 6.401
- Angle
- 0.0°
- Rate
- -2.590
- Best rate
- —
- Best angle
- —
The steepest direction is not asserted, it is found. Every direction is tried, and one of them wins. Which is why the step has a minus
The arrow points at the fastest way up. You want down, so you go the other way. That is the whole of the minus sign in the update rule.
Where the valley stops existing
At three knobs the picture is already hard; at thirteen thousand there is no picture, and the list of numbers takes over. Measuring those slopes one at a time would cost two runs of the machine each, twenty six thousand runs for a single step, which is why a faster way had to be found.
One slope per knob, written as a list. Every line asks the same question about a different knob: if I nudge this one, how much does the cost move?
Show it in symbols
gradient = [ slope for knob 1, slope for knob 2, … ]
The list is usually abbreviated to ∇L, and the step written w ← w − (step size × ∇L). That is the sentence above in symbols: go the opposite way to the list, by an amount you choose.
The misreading to head off
The gradient does not point at the minimum. It points at the steepest climb right here, which in a curved valley sends you into the opposite wall rather than along the floor.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.