Chapter 10
Embeddings
Meaning as a direction in space.
Models do not read words, they read coordinates. Where those coordinates come from, and why distance between them turns out to mean something.
- vectors
- similarity
- geometry
The image it is built onA long evening in a room where nobody wrote a seating plan. People drift, and by the end the ones who keep talking about the same things are standing together — not because anyone put them there, but because that is where the evening took them. And if you ask what the two ends of the room mean, nobody can tell you. Where it stops being true: people choose where they stand, and these numbers are pushed.
Everything so far has run on numbers that arrived as numbers. A word is not one. This chapter is where words become something the machinery of the last nine chapters can actually take in, and why the result turns out to mean something nobody put there.
Nothing runs on words
The machinery of the previous chapters multiplies and adds. A word cannot enter it, so something has to turn words into numbers first.
A row of numbers per word
Give every word a short list of numbers. Not a code, not a lookup into a dictionary of meanings: just a row of numbers, as many as you decide to use, sitting in a table with one row per word. The row is the word, as far as anything downstream is concerned.
A row of numbers like this is usually called a vector, and a stack of them a tensor. Both words are worth recognising and neither is needed here: it is a row of numbers, and calling it that costs nothing.
Nobody chooses the numbers
This is the question the table above raises and cannot answer by itself: where do those numbers come from? Who decided that cat should be −0.42 in its first column?
Nobody did. At the start every row is filled with random numbers and the table means nothing at all. Then it is treated exactly like every other parameter in the model: the loss is measured, the blame is handed back through the network by chapter 8, and the rows are nudged the way chapter 4 nudges anything. A row of numbers for a word is not a special kind of thing; it is a few more parameters, learnt in the same sweep as the weights.
So the honest answer to how you choose the numbers is that you do not. You choose how many there are, and the training does the rest.
Letters first, they are smaller
Do it on the twenty six letters instead, with two numbers each, and the whole table fits on screen. Train it and the vowels drift into one corner and the consonants into another, because those are the letters that behave alike. Nobody wrote down what a vowel is.
Training moves the words
The positions are parameters like any other, so they are learnt. Words used in the same way drift together, because that is what lowers the loss.
Press Train and watch. Then press New start and watch the same groups form from a different random beginning.
- steps
- 0
- loss
- 106.2
- words
- 18
- reading
- —
These positions are random. They mean nothing at all yet.
Distance ends up meaning something
Nobody defined similarity. It appears as a side effect, which is the part worth showing rather than asserting.
A word becomes a row of numbers, the row starts as noise, and the same descent that fits a line fits these too. What comes out is an arrangement in which distance means something, which no one designed and no one can fully read.
One word at a time is not a sentence. The next chapter is what reads a whole one, and what writes a whole one back.
The misreading to head off
The axes carry no meaning on their own. Looking for the dimension that encodes gender or colour is looking for something the training never promised to build.