How models learn
← The series

Chapter 12

Inference

Using what was learned.

Training is over and the settings are frozen. What runs now is half of what ran before, and the missing half explains most of what you notice while you wait for an answer.

  • forward pass
  • one token at a time
  • memory

The image it is built onSomeone who has finished their studies. They are not learning any more, they are answering, and they answer one word at a time.

Everything in the eleven chapters before this one was about getting the parameters right. They are right now, or as right as they are going to get, and they are frozen. What runs from here is half of what ran before, and the missing half explains most of what you notice while you wait.

Half the machine goes quiet

Training had three moments: the forward pass, the backward pass, and the adjustment of the parameters. Answering has only the first. The parameters are frozen and nothing is measured against a right answer, because there is no right answer to hand.

forward, work out the answerbackward, not run any morethe answernothing to compare it with
The two traversals of chapter 8, and only the lower one still runs. There is no truth to compare the answer with, so there is nothing to hand back along the upper one, and it is drawn as the thing that has stopped.

Context in, one token out

The model is given everything written so far and produces one piece of a word, the one it finds most likely to come next. That piece is added to what was written, and the whole thing goes back in. Everything you watch appear on screen is that loop.

Press Play, then watch the second line of the readout: how many pieces have been read to write the ones above.

everything written so far, read again from the start on every turnThecatsatonthethe model, unchangedmatone piece, the most likely nextit joins the context
pieces written
0
pieces read to write them
0

The model is not changed by any of this. Only the row above it grows.

Press play. The context grows by one piece each turn, and the model is run again from the top every single time.

Two ways of being used

A conversation, where the context keeps growing turn after turn, and a single request, where it all arrives at once. The mechanism underneath is the same loop in both cases; only the length of what goes in is different.

Why the first word is slow

The model reads the whole context before it can answer at all, so the wait before the first word grows with the length of what you asked. After that the words come out at a steady rate that does not depend on that length. The pause and the pace are two different things.

Drag the length to its maximum, and look at the bars after the first.

one request, drawn on a time line that does not rescalereading everything you askedthen one piece, then the next, at a steady pace0s1s2s3s4s
wait before the first piece
1.61 s
pieces per second after that
23.8

Drag the length up. The first bar stretches and the ones after it stay exactly where they were.

Drag the length of the request. The first bar stretches, the ones after it do not.

It does not read twice

Producing the hundredth piece would mean reading the ninety nine before it again, and the total work would grow like the square of the text. So the model keeps what it worked out on the way past, and reads only the new piece each turn. That is the difference between a wall and a straight line.

One pass forward, one piece of a word, and round again, on parameters that never move. That is the whole of what happens between your question and the answer arriving, and every part of it was built in one of the eleven chapters before this one.

The misreading to head off

A model that is answering is not learning. Nothing that happens during a conversation changes its parameters; what makes it look otherwise is the context getting longer, not the model getting different.