Chapter 09 · Not written yet
Backpropagation
Who is to blame, and by how much.
One line has two knobs. A network has millions, stacked in layers. Backpropagation is the bookkeeping that tells each one how much it contributed to the error.
- chain rule
- layers
- credit assignment
One neuron, differentiated by hand
Forward, each node computes a value from the ones behind it. Backward, starting from the output with a gradient of 1, every edge passes on how much its own input mattered. That is the local derivative, multiplied by whatever arrived. Doing that everywhere, in the right order, is the whole of backpropagation.
Nothing computed yet
How this chapter will go
Push the blame backwards by hand through a small network, and say what the backward pass computes.
The image it is built onEvery training example is a customer with a complaint. This output should be higher, that one lower. The complaints travel back through the shop and are added up.
One wrong answer, one demand
The output layer is not what it should be. Draw what this single example wants: some values up, some down, by different amounts.
Three ways to satisfy it
Change the offset, change the weights, or change what arrives from the layer before. Only the third one travels, and that is why there is an algorithm here at all.
Pull harder on the strong connections
A connection whose input was large has more effect, so it takes more of the blame. This single sentence is most of backpropagation.
The demand becomes a new demand
What the output layer wants from the layer before it is itself a list of wishes, and the same reasoning applies again. Repeat until the first layer.
Two traversals, never mixed. Forward computes an answer, backward hands out the blame, and the thick edges are the ones that carry most of it. Many examples at once
Every example asks for something different. Add the demands together and the sum is the gradient of the previous chapter, arrived at from the other end.
The misreading to head off
Backpropagation changes nothing. It works out how much each knob is at fault, and the step that follows is what actually moves them. Keeping the two apart is what makes the training loop readable.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.