Chapter 11 · Not written yet
Learning by heart
When a model answers without understanding.
A curve through every point is perfect on what it has seen and useless on anything else. Which is why some examples are kept back, and never shown.
- generalisation
- overfitting
- held-out data
How this chapter will go
Recognise a model that has memorised rather than understood, and say why examples are held back.
The image it is built onThe student who learns past papers by heart. Flawless on the questions that already came up, lost on the day.
Two candidates
A cloud of points, a simple curve that misses several of them, and a wandering curve that passes through every single one.
HELD-OUT ERROR IS LOWEST AT DEGREE 6 · VERTICAL SCALE COMPRESSED SO THAT 0 AND THE BLOW-UP BOTH FIT
- Degree
- 3
- Coefficients
- 4
- Training error
- 0.210
- Held-out error
- 0.307
- Best degree
- 6
- Verdict
- generalising
Pick one before reading on. The question is deliberately asked before the evidence is shown. The evidence arrives
New points appear, from the same source, never shown to either model. The wandering curve collapses and the simple one holds.
Two curves that separate
Plot the cost on the training examples and on the held back ones together. They fall together, then they part, and the moment they part is the whole diagnosis.
Three remedies, one line each
More examples, fewer knobs, or a force that pulls the knobs towards zero. The third one is a small extra term added to the cost, which is where regularisation comes from.
The misreading to head off
Overfitting is not caused by training for too long. It comes from a model with more freedom than the data can pin down, which is why removing examples produces it just as reliably as adding steps.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.