Chapter 14 · Not written yet
Encoder and decoder
Squeezing something down, then building it back up.
Read the whole input, compress it to a representation, generate an output one piece at a time. The shape behind translation, captioning and most of what people now call generative.
- attention
- sequences
- bottleneck
How this chapter will go
Follow a sentence through the squeeze and back out, and say what the bottleneck costs.
The image it is built onReading a page, closing the book, and writing it out in another language from memory alone.
Two jobs, not one
Something has to read the whole input, and something else has to produce an output one piece at a time. Splitting the two is the entire architecture.
The squeeze
The reader ends with a fixed amount of room, whatever the length of the input. The width of the drawing says the compression better than any sentence.
- came back as it went in
- came back wrong
- the room it all has to fit in
how much goes innumbers in the middle- kept
- 40
- went in
- 60
- recovered
- 9/10
- first third
- 2/3
- last third
- 3/3
- squeeze
- 1.5×
Hold the room still and lengthen the row. The far end survives; the beginning is what goes.
The throat in the middle is the room, drawn at its real width. Lengthen the row above it and the throat does not move; watch which end of the output goes first. Writing one piece at a time
The writer produces a word, looks at what it has already written, and produces the next. Its own output is part of its input.
What the bottleneck costs
A long input squeezed into a fixed space loses detail, and the loss is worse the further back it happened. Attention exists as an answer to exactly this.
The misreading to head off
Generative does not mean the machine invented anything. It produced the next piece, then the next, each one conditioned on everything before it.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.