What a neural network is, step by step

Here a unit has two inputs, 0.5 and 1.0, and two weights, 2 and -1. Each input is multiplied by its own weight to give its contribution. The first input is 0.5 with weight 2, so it contributes 1.0. The second input is 1.0 with weight -1, so it contributes -1.0. A positive weight means that input pushes the total up; a negative weight means it pushes the total down; a weight near zero means the input hardly matters. The bias line is used in the next stage.

Work out both contributions on paper, then run.

After adding the contributions, the unit adds one more number, its bias. The bias is a starting nudge that does not depend on the inputs, like a head start or a handicap. It lets a unit lean towards a high or low total even when the inputs are small. Here the two contributions cancel out (1.0 plus -1.0 is 0), so the whole total, called the weighted sum, is just the bias, 0.2. Training adjusts biases as well as weights; both count as the unit's learned numbers.

Run and trace the sum: 1.0, plus -1.0, plus 0.2.

The last thing a unit does is pass its total through a simple rule. The most common rule is called ReLU: keep the total if it is positive, and replace it with zero if it is negative. In Python that is just max(0, total). You can read it as: the unit reports how strongly its pattern is present, or stays silent. The result is called the unit's activation. This bend at zero matters more than it looks. Without it, stacking many units would add up to one big weighted sum, which can only describe very simple patterns. With it, many units together can describe the curved and broken shapes that handwriting needs.

Run and compare a positive total with the negative example -0.7.

Real units have many inputs: a unit reading a digit has 64. Writing out 64 multiplications would be silly, so NumPy has a shortcut. With two arrays of the same length, inputs @ weights multiplies each pair of matching items and adds the results. That operation is called a dot product, and it is exactly stages 1 and 2 without the bias. np.maximum(0, ...) is ReLU for arrays. The calculation has not changed; it is only shorter to write.

Run and check that the answer matches the long version from stage 3.

The output layer gives one raw score per answer, called a logit. Here there are three answers, the digits 2, 5, and 8, with scores 1, 2, and -1. Raw scores can be any size and can be negative, so they are turned into chances with a recipe called softmax:

1. Subtract the largest score from every score. This is only a safety step that keeps the numbers small; it does not change the final result.
2. Apply np.exp, which raises the number e (about 2.718) to the power of each score. You do not need to know e. What matters is that the result is always positive, and each extra point of score makes the result about 2.7 times bigger, so the leading answer pulls ahead.
3. Divide each result by their total, so the chances add up to 1.

The list of chances is called a distribution, the same word Level 1 used for next-word chances. The digit with the largest chance is the prediction. np.argmax finds its position, and classes turns that position into the digit it stands for.

Predict which digit wins before running.

For one training image, look at the chance the network gave to the correct digit. The loss is -math.log(chance). math.log is the natural logarithm, a function you do not need to calculate by hand; the table this program prints shows everything you need. When the chance for the right answer is high, such as 0.9, the loss is small. As the chance falls towards zero, the loss grows fast. The minus sign only makes the result positive. This loss is called cross-entropy. It rewards the network for being confident and right, and punishes it hardest for being confident and wrong. Accuracy only asks which digit won; loss also notices how sure the network was, which is why the two can move differently.

Predict the order of the four losses, smallest to largest, before running.

This is the smallest possible model: one weight, one input of 2.0, and a correct answer of 1.0. The weight starts at 0, so the prediction is 0 times 2, which is 0, and the squared-error loss from Level 1 is (0 minus 1) squared, which is 1. The gradient line is a formula that comes from a branch of mathematics called calculus; you do not need to work it out, only to read what it tells you. Its sign says which way to move the weight: negative means increase it. Its size says how steeply the loss changes. The update line moves the weight a small step in the direction that lowers the loss: the learning rate, 0.1, times the gradient, -4, gives -0.4, and subtracting -0.4 from 0 gives a new weight of 0.4. With the new weight the prediction is 0.8 and the loss drops from 1 to about 0.04.

Run and follow the numbers: gradient, new weight, loss before, loss after.

Now count the learned numbers in the network you will train next. Each of the 32 hidden units has 64 weights, one per pixel, plus one bias: 64 times 32 plus 32 is 2,080. Each of the 10 output units has 32 weights, one per hidden unit, plus one bias: 32 times 10 plus 10 is 330. Together that is 2,410 numbers for training to adjust. A bigger network has more numbers, which lets it describe more complicated patterns but also takes longer to train and can memorise its training examples. Only measuring on development data tells you whether bigger helps.

Check the two multiplications by hand, then run.