How models learn
← The series

Chapter 12

Inference

Using what was learned.

Training is over and the settings are frozen. What runs now is half of what ran before, and the missing half explains most of what you notice while you wait for an answer.

  • forward pass
  • one token at a time
  • memory

The image it is built onSomeone who has finished their studies. They are not learning any more, they are answering, and they answer one word at a time.

  1. Half the machine goes quiet

    Training had three moments: the forward pass, the backward pass, and the adjustment of the knobs. Answering has only the first. The knobs are frozen and nothing is measured against a right answer, because there is no right answer to hand.

    forward, work out the answerbackward, hand out the blametoo lowright
    The two traversals of chapter 8, drawn together again. Only the lower one still runs here. Nothing is comparing the answer with a truth, so there is no blame to hand back along the upper one.

  2. Context in, one token out

    The model is given everything written so far and produces one piece of a word, the one it finds most likely to come next. That piece is added to what was written, and the whole thing goes back in. Everything you watch appear on screen is that loop.

    everything written so far, read again from the start on every turnThecatsatonthethe model, unchangedmatone piece, the most likely nextit joins the context
    pieces written
    0
    pieces read to write them
    0

    The model is not changed by any of this. Only the row above it grows.

    Press play. The context grows by one piece each turn, and the model is run again from the top every single time.

  3. Two ways of being used

    A conversation, where the context keeps growing turn after turn, and a single request, where it all arrives at once. The mechanism underneath is the same loop in both cases; only the length of what goes in is different.

  4. Why the first word is slow

    The model reads the whole context before it can answer at all, so the wait before the first word grows with the length of what you asked. After that the words come out at a steady rate that does not depend on that length. The pause and the pace are two different things.

    one request, drawn on a time line that does not rescalereading everything you askedthen one piece, then the next, at a steady pace0s1s2s3s4s
    wait before the first piece
    1.61 s
    pieces per second after that
    23.8

    Drag the length up. The first bar stretches and the ones after it stay exactly where they were.

    Drag the length of the request. The first bar stretches, the ones after it do not.

  5. It does not read twice

    Producing the hundredth piece would mean reading the ninety nine before it again, and the total work would grow like the square of the text. So the model keeps what it worked out on the way past, and reads only the new piece each turn. That is the difference between a wall and a straight line.

The misreading to head off

A model that is answering is not learning. Nothing that happens during a conversation changes its knobs; what makes it look otherwise is the context getting longer, not the model getting different.