How models learn
← The series

Chapter 05

Batches

Whole set, small batch, and the examples kept back.

Looking at every example before each step is exact and slow. A handful is noisy and quick, and the noise turns out to be useful. A tenth of the data never enters any of it.

  • batches
  • noise
  • held-out data

The image it is built onA careful walker who surveys the whole valley before each step, against someone half cut who stumbles downhill quickly. The second one usually gets there first.

  1. The exact step is expensive

    A true gradient needs every example measured before a single knob moves. Put a number on it: with a million examples that is a million measurements per step.

  2. A handful already knows

    Take thirty examples at random and the direction they suggest is roughly the true one. Roughly is enough, because you are about to take another step anyway.

  3. The trade, on one slider

    Large batches give a smooth path and few steps per minute. Small batches give a ragged path and many. The slider moves between the two and the paths are drawn together.

    POSITIONHEIGHTDEEPEST FLOORA LESSER FLOOR
    • Every example · 100
    • A batch of 4, drawn afresh each step
    4
    1.00
    Batch
    4 / 100
    Step
    0 / 80
    Cost, batch
    0.1658
    Cost, full
    0.1658
    Batch error
    0.0000

    Both paths are waiting at the same starting point.

    Same landscape, same starting point. The smooth path is exact and slow, the ragged one is approximate and gets there first.

  4. Noise as an ally

    A ragged path does not settle into every small dip it meets. What looks like sloppiness is what keeps the search from stopping too early.

  5. Making the batches

    The examples are shuffled, then cut into packets of a fixed size, and one step is taken per packet. At the end of a pass they are shuffled again, so the same thirty never travel together twice.

  6. A tenth is never shown

    Before any of that cutting happens, ten examples in every hundred are set aside and kept out of training entirely. They are what you measure on afterwards, because a model can only be trusted on questions it has not already been given the answers to.

The misreading to head off

Noise is not a defect to be removed. It is a setting, and turning it to zero is a choice with its own cost.