How models learn
← The series

Chapter 07

The chain rule

Sensitivities multiply along a chain.

How much does the end move when the start moves? Down a train of gears the ratios multiply, and the steps of a calculation behave the same way.

  • sensitivity
  • composition
  • chain rule

The image it is built onA train of gears. If A turns three times faster than B, and B twice as fast as C, then A turns six times faster than C. The factors multiply. Where it stops being true: a gear has one fixed ratio, and a step of a calculation has a different sensitivity depending on where you are standing.

  1. What a sensitivity is

    Take one step of a calculation and nudge what goes into it. However much more the output moves than the input did, that factor is the sensitivity of the step. It is the same number as the slope of chapter 3, asked about a step instead of about a knob.

  2. Two gears, one ratio

    Turn the first and the second follows, faster or slower by a fixed factor. That factor is the sensitivity of this one link, and changing it changes the output speed with it.

  3. Three gears, one product

    Add a third and the total factor is the two multiplied. The rule is stated in gears before it is stated in anything else.

    × 3.00× 2.000.00 turns0.00 turnsyou turn here× 6.00 comes out
    • the crank you turn
    • the far end
    • the links between
    first ×
    3.00
    second ×
    2.00
    multiplied
    6.00
    if added
    5.00
    turns in
    0.00
    turns out
    0.00

    Press Turn it and watch the two ends, or drag the crank yourself.

    The factor from end to end is the product of the factors of each link. Set the two ratios and read the total.

  4. The machine is a train of gears

    Each step of the network is a gear: a small change on its input becomes a change on its output, multiplied by its sensitivity. So the sensitivity of the whole machine is the product of the sensitivities of its steps.

  5. One jammed gear stops everything

    If any link has a factor near zero, the whole product collapses and nothing reaches the far end. That is a gradient dying, seen before it is given a name.

Written on three links, with words rather than letters, so the product can be read aloud. Each bracket is the sensitivity of one gear: how much its output moves when its input moves.

Show it in symbols

total factor = (third gear) × (second gear) × (first gear)

With symbols instead of words, this is the chain rule. It says one thing: multiply along the chain.

The misreading to head off

Sensitivities multiply, they do not add. Two links that each double give four, not four by addition, and the figure should make that mistake visible rather than describe it.