Chapter 02
The loss
Every mistake, boiled down to one number.
An error is a gap, and a gap can be drawn as a square. Add the squares and you have the single number that the whole of training exists to push down.
- error
- squares
- loss
The image it is built onTasting the dish. One mouthful, one verdict, and the verdict is a number.
One example, one gap
The machine answers, the truth is known, and the gap between them is drawn as a vertical segment. That segment is the error on that example, and nothing else.
Why adding gaps fails
Add the gaps and a guess that is too high cancels one that is too low, so a bad model can score zero. Drag a point across the line and watch the total lie to you.
Turn each gap into a square
Give every segment a square of its own. A gap twice as large gets a square four times as big, so one bad answer costs more than two mediocre ones.
Drag the line to slide it, or either handle to tilt it. Arrow keys nudge; hold shift for more.
- Cost (total area)
- 17.81
- Slope
- -0.10
- Intercept
- 4.00
- Best possible
- 2.03
The squares sit under the plot rather than on it. A gap is a length in y, and a square needs a second direction: on the plot the only one going spare is the input axis, where a width would be claiming something untrue. Below, laid end to end, they only have to be the size they are, and the side they fall on says whether the guess was too low or too high. One number for the whole set
Add the areas and divide by how many there are. The result no longer depends on the number of examples, so two models can be compared on it. That number is the loss, and every chapter after this one exists to push it down.
What the number will not tell you
It says how much error there is, never where it is. Two very different models can share a loss, which is why the picture stays on screen next to it.
Written with symbols, using a hat on the letter to mean what the machine answered, and no hat for the truth:
Show it in symbols
loss = average of (ŷ − y)²
Nothing here is new. It is the picture above, written in a shorter way.
The misreading to head off
Squaring is not a mathematical trick to remove the minus sign. It is a decision about what counts as bad, and it makes one large mistake worse than several small ones.