How models learn
← The series

Chapter 06

Neural network

Neuron, layer, forward pass.

A neuron adds up what reaches it and squashes the result. Stack them and each layer folds the space a little more, until a number can be followed all the way from the input to the answer.

  • neuron
  • layers
  • activation

The image it is built onA chain of workshops. Each one takes what arrives, does its small transformation, and passes it on. No workshop sees the whole chain.

  1. One neuron, one sentence

    It adds up what reaches it, giving each input its own importance, then adds a number of its own, then squashes the result into a small range.

    x₁0.90x₂0.40x₃-0.70WEIGHTSSUM1.245b-0.65BIASSQUASH · TANH+1−1−6+6SUM, BEFORE SQUASHINGΣwxOUTPUT0.847THICKER EDGE = BIGGER WEIGHT · DASHED = NEGATIVE

    Inputs — what arrives

    0.90
    0.40
    -0.70

    Weights — how much each input matters

    1.25
    0.35
    -0.90

    Bias — the neuron’s own number

    -0.65
    w₁x₁
    1.125
    w₂x₂
    0.140
    w₃x₃
    0.630
    Bias
    -0.650
    Sum
    1.245
    Output
    0.847
    Slope
    0.283
    Three inputs, three weights, one bias and one squash. Every neuron in every network is this picture repeated.

  2. Why the squash is not optional

    Without it, stacking ten layers gives you exactly what one layer could do. Compose two straight lines and you get a straight line, so the depth would be free and useless.

  3. A layer is several of them

    Several neurons look at the same input with different weights, so each notices something different. That is all a layer is.

  4. Each layer folds the space

    Draw the input as a sheet of squared paper. The weights stretch and turn it, the squash bends it, and points that were tangled up on the way in end up on opposite sides of a straight cut. Depth is that fold applied again and again.

    one straight cut
    The first half turns the space, the second half folds it.
    • the band
    • everything else
    • the cut, once it works

    The band runs at an angle and its two outsides are separate pieces. No straight line has them apart.

    One layer, one fold. Drag the slider and watch the grid deform; the two clouds cannot be separated by a straight line on the left, and can be on the right.

  5. The forward pass

    A number enters, each layer transforms it, an answer comes out. Watch one value travel and light up the path it takes.

  6. Counting the knobs

    Count the weights and biases of this small network on screen. The number is already in the hundreds, which is where the word million starts to make sense.

The neuron adds what reaches it, weighting each input, then adds its own offset:

Show it in symbols

sum = w₁x₁ + w₂x₂ + … + wₙxₙ + b

That sum can be any number at all, and a bounded signal is wanted, so it is squashed: output = squash(sum). The word activation is for later.

The misreading to head off

The word neuron borrows from biology and stops there. The relation is the one between a bird and an aeroplane: the same problem, solved twice, differently.