Chapter 06 · Not written yet
Fast, or straight
Whole batch, small batch, one at a time.
Looking at every example before each step is exact and slow. A handful is noisy and quick, and the noise turns out to be useful.
- batches
- noise
- momentum
How this chapter will go
Choose between the whole batch, a small batch and one example at a time, and say what the noise buys.
The image it is built onA careful walker who surveys the whole valley before each step, against someone half cut who stumbles downhill quickly. The second one usually gets there first.
The exact step is expensive
A true gradient needs every example measured before a single knob moves. Put a number on it: with a million examples that is a million measurements per step.
A handful already knows
Take thirty examples at random and the direction they suggest is roughly the true one. Roughly is enough, because you are about to take another step anyway.
The trade, on one slider
Large batches give a smooth path and few steps per minute. Small batches give a ragged path and many. The slider moves between the two and the paths are drawn together.
- Every example · 100
- A batch of 4, drawn afresh each step
- Batch
- 4 / 100
- Step
- 0 / 80
- Cost, batch
- 0.1658
- Cost, full
- 0.1658
- Batch error
- 0.0000
Both paths are waiting at the same starting point.
Same landscape, same starting point. The smooth path is exact and slow, the ragged one is approximate and gets there first. Noise as an ally
A ragged path does not settle into every small dip it meets. What looks like sloppiness is what keeps the search from stopping too early.
The misreading to head off
Noise is not a defect to be removed. It is a setting, and turning it to zero is a choice with its own cost.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.