How models learn
← The series

Chapter 02

The loss

Every mistake, boiled down to one number.

A mistake is the distance between what the machine answered and what was true, and that distance can be drawn as a square. Add the squares and you have the single number that the whole of training exists to push down.

  • error
  • squares
  • loss

The image it is built onTasting the dish. One mouthful, one verdict, and the verdict is a number.

One number has to stand for everything the machine got wrong. This chapter builds it.

One answer, one mistake

The machine answers, the truth is known, and the distance between the two is drawn as a vertical segment. That segment is the mistake made on that example, and nothing else.

Why adding them up fails

Add the mistakes and an answer that is too high cancels one that is too low, so a bad model can score zero. What is wanted is a way of counting every mistake as a positive amount, whichever side it falls on.

Give every mistake a square

Draw a square whose side is the distance. Its area is that distance multiplied by itself, so a mistake twice as large counts four times as much, and one bad answer costs more than two mediocre ones.

Add the areas and divide by how many there are. The result no longer depends on how many examples you happened to measure, so two models can be compared on it. That number is the loss.

Drag the line and watch the squares under it grow and shrink. Then press Best fit and see how small they can get.

EACH LINE IS ONE MISTAKETHE SAME MISTAKES, SQUARED AND LAID END TO ENDGUESS TOO LOWGUESS TOO HIGH

Drag the line to slide it, or either handle to tilt it. Arrow keys nudge; hold shift for more.

Loss (total area)
17.81
Slope
-0.10
Intercept
4.00
Best possible
2.03
The squares sit under the plot rather than on it. A mistake is a length in y, and a square needs a second direction: on the plot the only one going spare is the input axis, where a width would be claiming something untrue. Below, laid end to end, they only have to be the size they are, and the side they fall on says whether the guess was too low or too high.

What the number will not tell you

It says how much error there is, never where it is. Two very different models can share a loss, which is why the picture stays on screen next to it.

The loss

Written with symbols. y is the true value, ŷ is what the machine answered, so ŷ − y is the mistake on one example. Square it, and average over all of them:

loss=average of (y^−y)2\text{loss} = \text{average of } (\hat{y} - y)^2

Nothing here is new. It is the picture above, written in a shorter way.

The loss is one number that stands for every mistake at once, and training is the act of pushing it down.

It says how wrong the machine is. It does not say which parameter to change, in which direction, or by how much. That is the next chapter.

The misreading to head off

Squaring is not a mathematical trick to remove the minus sign. It is a decision about what counts as bad, and it makes one large mistake worse than several small ones.