Chapter 06

Neural network

Neuron, layer, forward pass.

A neuron adds up what reaches it and squashes the result. Stack them and each layer folds the space a little more, until a number can be followed all the way from the input to the answer.

neuron
layers
activation

The image it is built onA chain of workshops. Each one takes what arrives, does its small transformation, and passes it on. No workshop sees the whole chain.

Everything so far has been a line with two parameters. This is the machine those parameters actually live in.

the path one value takes
every other connection

layers: 4
neurons: 29
weights: 172
biases: 29
parameters: 201

Every line is a weight, every tick a neuron with a bias of its own. Press Run to send a value through.

One input, four layers, one output. Every faint line is a weight and every tick is a neuron with an offset of its own, which is where the parameter count comes from.

Press Run and watch one value cross the whole machine. The bright path is the route it takes; all the pale lines are the routes it does not.

One neuron, one sentence

It adds up what reaches it, giving each input its own importance, then adds a number of its own, then squashes the result into a small range.

That importance is a weight. An input x arrives, it is multiplied by the weight w that belongs to that connection, and w·x is what the neuron actually receives from it. The weight is one parameter, and on the first layer it is one of the numbers the whole of chapter 3 was about nudging.

Drag a weight to zero and watch its input stop mattering. Then drag it negative and watch the same input start arguing the other way.

Inputs: what arrives

x₁0.90

x₂0.40

x₃-0.70

Weights: how much each input matters

w₁1.25

w₂0.35

w₃-0.90

Bias: the neuron’s own number

b-0.65

w₁x₁: 1.125
w₂x₂: 0.140
w₃x₃: 0.630
Bias: -0.650
Sum: 1.245
Output: 0.847
Slope: 0.283

Three inputs, three weights, one bias and one squash. Every neuron in every network is this picture repeated.

Why the squash is not optional

Without it, stacking ten layers gives you exactly what one layer could do. Compose two straight lines and you get a straight line, so the depth would be free and useless.

A layer is several of them, and depth is layers of layers

Several neurons look at the same input with different weights, so each notices something different. That is all a layer is.

Put one layer after another and each works on what the one before it produced, not on the original input. A number enters, every layer transforms it in turn, an answer comes out the far end. That crossing is called the forward pass, and it is what the drawing at the top of this chapter is doing when you press Run.

What a layer does to the space, and what a second one adds

Draw the input as a sheet of squared paper. The weights stretch and turn it, the squash bends it, and points that were tangled up on the way in end up on opposite sides of a straight cut.

One fold is often not enough, and this is where depth stops being a slogan. The data below needs two: after the first layer the two colours are still interleaved, and no straight line does better than about three quarters of them. The second layer slides the plane and folds it again, and then one straight cut gets every point right. That is the whole answer to what another layer is for.

Press Fold it and watch both layers run. Then stop the slider at exactly one layer and try to find a vertical line that separates the colours; there is not one.

layers appliedEach layer turns or slides the space, then folds it. Two of them are needed here.

the band
everything else
the cut, once it works

Three bands at an angle, alternating. No straight line has the two colours apart, and folding once will not be enough either.

Three bands of points, alternating, and two layers applied one after the other. The cut appears only once it is true, which is not until the second fold has closed.

Counting the parameters

Go back to the drawing at the top and read the numbers under it. That small network, eight neurons wide at most, already has two hundred parameters. A real one has billions, and nothing about the picture changes except how many faint lines there are.

One neuron

The neuron adds what reaches it, weighting each input, then adds its own offset. Here xᵢ is an input, wᵢ the weight on that connection, and b the offset:

\text{sum} = w_1 x_1 + w_2 x_2 + \dots + w_n x_n + b

That sum can be any number at all, and a bounded signal is wanted, so it is squashed: output = squash(sum). Every wᵢ and every b in the network is a parameter, and that is the whole of what training changes.

A neuron weighs its inputs, adds an offset and squashes. A layer is several of those. A network is layers of layers, and its parameters are every weight and every offset in it.

Chapter 3 nudged one parameter at a time to find its slope. With two hundred of them, and then with billions, that will not do. The next two chapters are about getting every slope at once.

The misreading to head off

The word neuron borrows from biology and stops there. The relation is the one between a bird and an aeroplane: the same problem, solved twice, differently.