Chapter 07
The chain rule
Sensitivities multiply along a chain.
How much does the end move when the start moves? Down a train of gears the ratios multiply, and the steps of a calculation behave the same way.
- sensitivity
- composition
- chain rule
The image it is built onA train of gears. If A turns three times faster than B, and B twice as fast as C, then A turns six times faster than C. The factors multiply. Where it stops being true: a gear has one fixed ratio, and a step of a calculation has a different sensitivity depending on where you are standing.
A network is a chain. The input becomes something, that becomes something else, and eventually something is the answer. To improve any one parameter you have to know how much the far end of that chain moves when you nudge it, and there is exactly one rule for that.
What a sensitivity is
Take one step of a calculation and nudge what goes into it. However much more the output moves than the input did, that factor is the sensitivity of the step. It is the same number as the slope of chapter 3, asked about a step instead of about a parameter.
Two gears, one ratio
Turn the first and the second follows, faster or slower by a fixed factor. That factor is the sensitivity of this one link, and changing it changes the output speed with it.
Three gears, one product
Add a third and the total factor is the two multiplied. The rule is stated in gears before it is stated in anything else.
Set both ratios to 2 and read the two numbers in the panel. Then set them to 3 and 2, and read them again.
- the crank you turn
- the far end
- the links between
- first ×
- 3.00
- second ×
- 2.00
- multiplied
- 6.00
- if added
- 5.00
- turns in
- 0.00
- turns out
- 0.00
Press Turn it and watch the two ends, or drag the crank yourself.
Every step of a network is a gear
A layer does one thing to what reaches it: multiply by its weights, add its offset, squash the result. That is a step, and like any step it has a sensitivity — nudge what goes in, and what comes out moves by some factor.
Now look at a weight buried in the first layer. Nothing connects it to the loss directly. What it touches is the first layer’s output, which goes into the second layer, whose output goes into the third, and so on until a number comes out and is compared with the right answer. Every one of those steps sits between that weight and the loss.
So the sensitivity of the loss to that one weight is the product of the sensitivities of every step along the way. Not a new rule for networks: the same rule as the gears, on a chain that happens to be forty links long instead of three. This is why the chain rule is the chapter before backpropagation, and not a detour.
One jammed gear stops everything
If any link in that chain has a factor near zero, the whole product collapses and nothing reaches the far end. A weight fifty layers deep can be enormously important to the answer and still be told, by the product, that it barely matters. That is a gradient dying, seen before it is given a name.
Press Jam a link on the train above and watch the far end stop.
Written on three links, with words rather than letters, so the product can be read aloud. Each bracket is the sensitivity of one gear: how much its output moves when its input moves.
With symbols instead of words, this is the chain rule. It says one thing: multiply along the chain.
A sensitivity is a factor, factors along a chain multiply, and a network is a chain. That is the whole of it, and it is enough to say what any parameter anywhere in a network is worth to the loss.
Saying it is not the same as working it out. A network has millions of these chains and they overlap; the next chapter is how you get every one of them in a single sweep.
The misreading to head off
Sensitivities multiply, they do not add. Two links that each double give four, not four by addition, and the figure should make that mistake visible rather than describe it.