How models learn
← The series

Chapter 07 · Not written yet

A machine in layers

Neuron, layer, forward pass.

A neuron adds up what reaches it and squashes the result. Stack them and a number can be followed all the way from the input to the answer.

  • neuron
  • layers
  • activation

How this chapter will go

Follow one number from the input to the answer, and say in a sentence what a single neuron does.

The image it is built onA chain of workshops. Each one takes what arrives, does its small transformation, and passes it on. No workshop sees the whole chain.

  1. One neuron, one sentence

    It adds up what reaches it, giving each input its own importance, then adds a number of its own, then squashes the result into a small range.

    x₁0.90x₂0.40x₃-0.70WEIGHTSSUM1.245b-0.65BIASSQUASH · TANH+1−1−6+6SUM, BEFORE SQUASHINGΣwxOUTPUT0.847THICKER EDGE = BIGGER WEIGHT · DASHED = NEGATIVE

    Inputs — what arrives

    0.90
    0.40
    -0.70

    Weights — how much each input matters

    1.25
    0.35
    -0.90

    Bias — the neuron’s own number

    -0.65
    w₁x₁
    1.125
    w₂x₂
    0.140
    w₃x₃
    0.630
    Bias
    -0.650
    Sum
    1.245
    Output
    0.847
    Slope
    0.283
    Three inputs, three weights, one bias and one squash. Every neuron in every network is this picture repeated.

  2. Why the squash is not optional

    Without it, stacking ten layers gives you exactly what one layer could do. Compose two straight lines and you get a straight line, so the depth would be free and useless.

  3. A layer is several of them

    Several neurons look at the same input with different weights, so each notices something different. That is all a layer is.

  4. The forward pass

    A number enters, each layer transforms it, an answer comes out. Watch one value travel and light up the path it takes.

  5. Counting the knobs

    Count the weights and biases of this small network on screen. The number is already in the hundreds, which is where the word million starts to make sense.

The neuron adds what reaches it, weighting each input, then adds its own offset:

Show it in symbols

sum = w₁x₁ + w₂x₂ + … + wₙxₙ + b

That sum can be any number at all, and a bounded signal is wanted, so it is squashed: output = squash(sum). The word activation is for later.

The misreading to head off

The word neuron borrows from biology and stops there. The relation is the one between a bird and an aeroplane: the same problem, solved twice, differently.

There is no date, no list to join and nothing to subscribe to. The chapter simply appears here once the drawing works.