Chapter 08
Backpropagation
Who is to blame, and by how much.
One line has two parameters. A network has millions, stacked in layers. Backpropagation is the bookkeeping that tells each one how much it contributed to the error.
- chain rule
- layers
- credit assignment
One neuron, differentiated by hand
Forward, each node computes a value from the ones behind it. Backward, starting from the output with a gradient of 1, every edge passes on how much its own input mattered. That is the local derivative, multiplied by whatever arrived. Doing that everywhere, in the right order, is the whole of backpropagation.
Press Step until the pale pulse has crossed the graph, then keep pressing: the accent one comes back the other way.
Nothing computed yet
- top half · the value it worked out on the way there
- bottom half · the gradient, filled in on the way back
- on each edge · the local derivative, times what arrived
The image it is built onEvery training example is a customer with a complaint. This output should be higher, that one lower. The complaints travel back through the shop and are added up.
The last chapter said what a parameter is worth to the loss: the product of the sensitivities between it and the end. A network has millions of parameters and working them out one at a time would take a lifetime per step. This is how you get all of them in one sweep.
One wrong answer, one demand
The output layer is not what it should be. Draw what this single example wants: some values up, some down, by different amounts.
Three ways to satisfy it
Change the offset, change the weights, or change what arrives from the layer before. Only the third one travels, and that is why there is an algorithm here at all.
Pull harder on the strong connections
A connection whose input was large has more effect, so it takes more of the blame. This single sentence is most of backpropagation.
The demand becomes a new demand
What the output layer wants from the layer before it is itself a list of wishes, and the same reasoning applies again. Repeat until the first layer.
Why it is done backwards
Nudging one parameter and re-running the machine to see what happens would cost one full run per parameter, so millions of runs for a single step. Going backwards gets every parameter its share of the blame in one traversal, which is the whole reason this chapter exists.
It reads, and moves nothing
Watch what the backward pass actually leaves behind: a number beside each parameter, saying how much that parameter is at fault. Not a new value for the parameter. Nothing in the network is any different at the end of the pass than it was at the start, and the answer it would give is the same answer. Reading and changing are two jobs, and this one is the reading.
Step the graph at the top to the end. Every value in the upper half is the same as it was; only the lower halves have filled in.
Many examples at once
Every example asks for something different. Add the demands together and the sum is the gradient of chapter 3, arrived at from the other end.
One sweep forward to get an answer, one sweep back to find out who is responsible for it being wrong. At the end of the two you have a number for every parameter in the network, and not one of them has moved.
Moving them is the next chapter, along with everything else that has to happen in order, over and over, for a model to end up any good.
The misreading to head off
Backpropagation reads; it does not move anything. It works out how much each parameter is at fault, writes that number down beside it, and stops there. The step that follows is what actually changes them, and keeping the two apart is what makes the training loop readable.