Chapter 12 · Not written yet
Embeddings
Meaning as a direction in space.
Models do not read words, they read coordinates. Where those coordinates come from, and why distance between them turns out to mean something.
- vectors
- similarity
- geometry
How this chapter will go
Say why a word becomes a point, and what the distance between two points is doing.
The image it is built onA map whose axes have no names. Nobody chose where the towns go, and yet towns that belong together end up close.
Nothing runs on words
The machinery of the previous chapters multiplies and adds. A word cannot enter it, so something has to turn words into numbers first.
A row of numbers per word
Give every word a short list of numbers, which is to say a position. At the start the positions are random and mean nothing.
Training moves the towns
The positions are knobs like any other, so they are learnt. Words used in the same way drift together, because that is what lowers the cost.
steps taken per frame- steps
- 0
- cost
- 22.8
- words
- 18
- reading
- —
These positions are random. They mean nothing at all yet.
Eighteen words, starting nowhere in particular. Nothing here was placed by hand, and pressing New start gives a different random beginning that arrives at the same neighbourhoods. Distance ends up meaning something
Nobody defined similarity. It appears as a side effect, which is the part worth showing rather than asserting.
The misreading to head off
The axes carry no meaning on their own. Looking for the dimension that encodes gender or colour is looking for something the training never promised to build.
There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.