Chapter 05
Batches
Whole set, small batch, and the examples kept back.
Looking at every example before each step is exact and slow. A handful is noisy and quick, and the noise turns out to be useful. A tenth of the data never enters any of it.
- batches
- noise
- held-out data
The image it is built onA careful walker who surveys the whole valley before each step, against someone half cut who stumbles downhill quickly. The second one usually gets there first.
The gradient of chapter 3 is exact, and exact is expensive. This chapter is about paying less for it, and about what you get back for the inexactness.
The exact step is expensive
A true gradient needs every example measured before a single parameter moves. Put a number on it: with a million examples that is a million measurements, and that buys one step.
A handful already knows
Take thirty examples at random and the direction they suggest is roughly the true one. Roughly is enough, because you are about to take another step anyway, and the next thirty will be different.
So the trade is this. A large batch gives a smooth path and few steps for the examples you read. A small one gives a ragged path and many. Set the two sliders against each other and watch both at once.
Leave A at 10 and B at 80, press Run, and watch the two counters. Both walks take the same number of steps; only one of them has read eight times as much to do it.
- Batch of 10, drawn afresh each step
- Batch of 80
- Step
- 0 / 80
- Loss, A
- 0.1658
- Loss, B
- 0.1658
- Examples read, A
- 0
- Examples read, B
- 0
- Batch error, A
- 0.0000
- Batch error, B
- 0.0000
Both paths are waiting at the same starting point.
What the noise is, and why it helps
Nothing is being added to the path to make it wobble. Each step follows the gradient of the batch it drew, exactly, and a batch of ten disagrees with the full set simply because ten examples are not a hundred. That disagreement is the noise, and the figure measures it: batch error is the distance between the batch’s answer and the true one.
A path that wobbles does not settle into every small dip it meets. It gets shaken out again, and carries on down. The careful path has no such luck: it goes exactly downhill, so the first floor it reaches is the one it stops on, and if a deeper one lay over the next ridge it will never know.
Set A to 1 and B to 100, press Run a few times, and watch which one more often ends up in the deeper basin.
Making the batches
The examples are shuffled, then cut into packets of a fixed size, and one step is taken per packet. At the end of a pass they are shuffled again, so the same thirty never travel together twice.
Training data and evaluation data
Before any of that cutting happens, ten examples in every hundred are set aside and never trained on. They are what you measure on afterwards, because a model can only be trusted on questions it has not already been given the answers to. Everything in this chapter happens on the other ninety.
A batch is a guess at the gradient, bought cheaply. The guess is wrong by a measurable amount, and being wrong by that amount is what keeps the search from stopping at the first floor it finds.
So far the machine has been a line with two parameters. The next chapter builds one with layers.
The misreading to head off
Noise is not a defect to be removed. It is a setting, and turning it to zero is a choice with its own cost.