How models learn
← The series

Chapter 11

Encoder and decoder

Squeezing something down, then building it back up.

Read the whole input, compress it to a representation, generate an output one piece at a time. The shape behind translation, captioning and most of what people now call generative.

  • sequences
  • bottleneck
  • attention

The image it is built onReading a page, closing the book, and writing it out in another language from memory alone.

Two halves: one that reads all of the input, one that writes the output a piece at a time. The trouble between them is that the first has to hand the second everything it learnt in a fixed amount of room, and the fix for that is the idea most of this field now runs on.

Two jobs, not one

Something has to read the whole input, and something else has to produce an output one piece at a time. Splitting the two is the entire architecture.

The squeeze

The reader ends with a fixed amount of room, whatever the length of the input. The width of the drawing says the compression better than any sentence.

Hold the room where it is, drag the length up, and press Write it out. Watch which end of the row comes back wrong.

goes in40 numbershowever long the rowcomes outthe front of the row fades first, because everything after it landed on top
  • came back as it went in
  • came back wrong
  • the room it all has to fit in
how much goes in
numbers in the middle
kept
40
went in
60
recovered
9/10
first third
2/3
last third
3/3
squeeze
1.5×

Hold the room still and lengthen the row. The far end survives; the beginning is what goes.

The throat in the middle is the room, drawn at its real width. Lengthen the row above it and the throat does not move; watch which end of the output goes first.

Writing one piece at a time

The writer produces a word, looks at what it has already written, and produces the next. Its own output is part of its input.

What the bottleneck costs

A long input squeezed into a fixed space loses detail, and the loss is worse the further back it happened. Closing the book was the mistake.

Leave the book open

So stop closing it. Let the writer look back at the input every time it produces a word, and let it decide each time which parts of the input are worth looking at. That is attention, and it is what removed the throat from the drawing. The rest of this chapter is how a machine decides where to look, because a decision is not a thing you can leave vague and still claim to have explained.

What each word asks, and what it offers

Give every word two more short rows of numbers, alongside the row that already is the word. Call the first one what this word is looking for, and the second what this word has to offer. Both are worked out from the word itself, by parameters learnt in the same sweep as everything else — nobody writes them.

The word it, halfway through a sentence, is looking for something that acts: a thing that could be the it. The word cat, sitting further back, has exactly that to offer. The word the has very little to offer anybody.

To find out how well two words go together, take one word’s looking-for and the other word’s offer and multiply them term by term, then add up. One number. Large when the two line up, small when they do not.

The match is the weight

Do that between the word being written and every word of the input, then scale the numbers so they add up to one across the row. Those are the weights: how much of each input word goes into this output word. Every number in the grid below is one of those sums, and not one of them was written down by anybody.

Click "it" down the left, read what it is asking for in the panel, then press Change one word.

thethecatanimalsatactiononthethethematsurfaceitpronounsleptactionthecatsatonthematitslepteach row looks along"it" is asking forsomething that acts · 1.5something being done · 0.4"cat" offerssomething that acts · 1.5asking, times offeringq · k = 1.30scaled so the row adds to oneweight = 74%brightness is the weight, on a fixed nought-to-one scale
how decisively the weights pick a winner
  • brightness · how much of that word goes into this one
  • the arcs · the same row, drawn over the sentence
  • under each word · the one thing about it the model was given
reading
it
most weight to
cat
that weight
0.74
row total
1.000
spread, bits
1.55

Nothing was written into this grid. Change one word and watch every number move.

Words down one side, words across the other, brightness for the weight. Each row is one word looking along the whole sentence. Change a word and watch the weights move.

One attention weight

Two rows of numbers per word, as above: what it asks for, and what it offers. The match between a word being written and a word of the input is those two multiplied term by term and added up. The weights are those matches scaled so that each row adds to one, which is what makes them shares of a whole rather than loose scores.

match(i,j)=qi⋅kjwij=e match(i,j)∑ne match(i,n)\text{match}(i,j) = q_i \cdot k_j \qquad\qquad w_{ij} = \frac{e^{\,\text{match}(i,j)}}{\sum_n e^{\,\text{match}(i,n)}}

The two rows have names you will meet everywhere: what a word asks for is its query, q, and what it offers is its key, k. The scaling step is called a softmax, and all it does is turn a row of numbers into a row of shares that add up to one.

Read it all, hold it, write it back one piece at a time, and at every piece decide afresh which parts of the input still matter. The deciding is a multiplication between what one word wants and what another has, and nothing in it was set by hand.

That is the machine, trained. The last chapter is the same machine with the training switched off, which is the one you have actually been talking to.

The misreading to head off

Generative does not mean the machine invented anything. It produced the next piece, then the next, each one conditioned on everything before it. And the attention weights are not an explanation: they show what was combined, not why the answer came out the way it did.