How a neural network reads
a handwritten digit

The drawing surface was never a picture. The 28×28 canvas is a grid of 784 cells, and each cell holds one number — a brightness from 0 (paper) to 255 (full ink). Your pen stroke lands anti-aliased: the core of the line saturates, but its edges trail off through greys like 120 and 12. Nothing downstream will ever see “a seven”, a stroke, or a shape. From here on, the network receives exactly 784 integers, and everything it will ever know about your drawing has to be reconstructed from them.

The network learned to read from MNIST — 60,000 training digits remixed in the 1990s from scanned US government forms, plus 10,000 held back for testing. Half were written by Census Bureau clerks, all tidy loops and even pressure; half by American high-school students, who wrote like American high-school students. NIST’s original split trained on the clerks and tested on the teenagers; Yann LeCun’s team shuffled the two together precisely because a network raised on neat handwriting falls apart on messy handwriting. Your scribble is being read by a machine educated on both.

Before a single digit reached the network, MNIST standardized it. Each one was scaled to fit a 20×20 box, then placed inside the 28×28 frame so that the center of mass of its ink — the average position of every dark pixel — sits exactly in the middle. Then every value is divided by 255, mapping the canvas into 0.0–1.0. None of this is glamorous, and all of it matters: training staggers on lopsided, unscaled inputs. A surprising share of machine learning is exactly this — janitorial work performed on numbers.

The grid was a courtesy for your eyes. The network flattens it, row after row, into a single list of 784 numbers — and geometrically, that list is one point in a 784-dimensional space. Every seven anyone has ever drawn is a point somewhere nearby; the eights live in a different neighborhood. “Recognizing digits” turns out to mean carving that space into ten regions and reporting which one your point fell into. Note the counter below: still zero. Re-encoding is free. Thinking is multiplication, and none has happened yet.

Meet neuron #47, one cell of the layer your seven is about to hit. It owns 784 weights — one per pixel — and a single bias. Its whole job: multiply every pixel by its matching weight, add everything up, add the bias: w·x + b = 2.40. That is 784 multiply-accumulates, and the counter in the panel below finally moves. There is nothing else inside — no rules, no templates, no if-statements. Everything the network will ever “know” lives in how these multipliers are set.

The design is a cartoon of a biological neuron: dendrites collect signals, the soma sums them, the axon fires the result forward. The original still embarrasses the copy — your brain runs 86 billion neurons on roughly 20 watts, a dim light bulb, while the datacenter GPU caricaturing a sliver of it pulls 700. The cartoon throws away nearly everything — spike timing, neurotransmitter chemistry, dendritic computation — and keeps one load-bearing idea: many weighted inputs, one output. That idea, it turns out, was the part that mattered.

The idea is older than the moon landing. In 1958, Frank Rosenblatt’s Mark I Perceptron read images through 400 photocells in a 20×20 grid — nearly the same retina as our 28×28 canvas — and stored its weights in 512 potentiometers physically turned by electric motors. Learning meant knobs rotating. The Navy’s press conference got away from everyone: the New York Times relayed the claim below with a straight face, sixty-eight years ago. The hype cycle is not new; only the hardware is.

One flaw would make all of this pointless: chaining weighted sums produces… another weighted sum. Stack a hundred purely linear layers and the tower collapses, mathematically, into one. So after summing, every neuron bends its output through an activation function. Ours is ReLU — max(0, x) — keep positives, zero out negatives. It looks too stupid to matter. But that one kink is the difference between a network that can only draw straight lines through 784-dimensional space and one that can carve it into any shape training demands.

So what did training actually put inside #47? Take its 784 weights and reshape them back into a 28×28 image: they draw a faint picture. Positive weights trace a horizontal band where a seven’s top bar tends to sit; negative weights punish ink where sevens rarely put any. A neuron is a feature detector, and its weights are literally a portrait of the feature it hunts. Your seven’s top bar lands square on the bright band, and #47 fires 0.83 — one vote, of 128 still to come, that this drawing has a bar on top.

The hidden layer is neuron #47’s whole cohort: 128 cells, each owning its own 784 weights, each reading your entire seven at the same instant. Each one hunts a different stroke. #12 looks for a closed loop (bad news for it today), #47 for the top bar, #91 for a long descending diagonal. One layer, one instant: 784 × 128 = 100,352 multiply-accumulates. The counter that crawled to 784 last chapter jumps three orders of magnitude in a single step — and this is still the small layer of a toy network.

Count everything adjustable: 784 × 128 weights plus 128 biases makes 100,480 parameters in the hidden layer; the output layer will add 1,290 more. Total: 101,770 knobs. The Mark I’s motors turned 512; this throwaway tutorial network carries 199 times as many, and a frontier language model carries roughly ten million times more still — the same multiply-add-bend, scaled until the count stops fitting in comprehension. Training, whenever it finally happens, means choosing a value for every single one of them.

Nothing stops you from stacking again. A second layer wouldn’t read pixels — it would read the 128 stroke-opinions, and detect combinations: “top bar AND long diagonal AND no loop anywhere.” That is depth’s whole trick: features eat features. Strokes become junctions, junctions become digit-parts, digit-parts become verdicts. And the uncomfortable, wonderful part — nobody writes any of it. The hierarchy precipitates out of training on its own, which is why these layers are called hidden: even their author doesn’t decide what they detect.

In 1989 George Cybenko proved the universal approximation theorem: a single hidden layer, wide enough, can approximate any continuous function — including “pixels → digit”. So why does anyone bother with depth? The catch is load-bearing: “wide enough” can mean absurdly, exponentially wide, and the theorem is pure existence — it never says how to find the weights, or how many examples finding them would take. A guarantee that a needle exists, silent about the haystack. Existence is not a recipe; deep learning is the recipe.

Here is what this whole chapter looks like to the machine: one line. h = x @ W₁ — a [1×784] vector times a [784×128] matrix, done on a laptop CPU in roughly 40 microseconds. This is why graphics cards conquered AI: GPUs were already engines for exactly this operation, built to push triangles through the same arithmetic. At around 10¹⁵ multiply-accumulates per second, one modern accelerator could in principle read five billion of your sevens every second. The forward pass isn’t clever. It is algebra with a very good publicist.

The output layer is ten neurons, one per digit, each reading all 128 hidden activations: 128 × 10 = 1,280 more multiply-accumulates, and the forward pass closes its books at 101,632. Their raw outputs are called logits — unbounded scores, not yet probabilities. Today’s tally: seven scores 11.9, one scores 5.2 (your long diagonal fooled nobody, but it tried), eight limps in at −3.1, its loop detector silent. Ten opinions on the table. Still no verdict — nothing here sums to anything meaningful yet.

Softmax converts scores into a bet: exponentiate every logit, divide by the sum. e^11.9 towers over e^5.2 by a factor of about 800, so the seven’s comfortable lead becomes a landslide — 98.3%, against 1.2% for the one, dust for everyone else. The exponential is the whole point: it manufactures decisiveness. And the formula isn’t native to computing at all. It is the Boltzmann distribution, built in the 19th century to describe how particles spread across energy states, now doing digit recognition under an assumed name.

Here is what 98.3% does not mean: a 98.3% chance of being right. It is the size of the network’s bet, and bets can be rigged. In 2015, Ian Goodfellow’s team added pixel noise carefully computed to be invisible to humans, and a state-of-the-art network reclassified a panda as a gibbon — at 99.3% confidence, higher than it had been about the panda. Softmax reports conviction, not truth; calibrating conviction against reality is a separate and still largely unsolved job. Keep an asterisk next to every confident number in this tutorial.

Total elapsed: about two milliseconds on a laptop CPU for all 101,632 multiply-accumulates. The best MNIST readers err on roughly 23 digits in 10,000 — matching estimates of careful humans. Before celebrating, run one experiment: nudge your seven two pixels left. Every one of the 101,770 weights now stares at the wrong ink — #47’s violet band misses your bar entirely — and the confidence readout stutters. The weights didn’t change. The architecture just revealed its worldview: it memorized where ink sits, not what ink looks like. Hold that thought for chapter 05.

Rewind to before training. The 101,770 weights are small random numbers, and the freshly hatched network reads your seven as… 10.2% seven, 9.8% one, 10.1% eight. A shrug, uniformly distributed. Every ability the last three chapters showed off — the stroke detectors, the landslide verdict — was absent at birth. Randomness is the blank page training writes on, and it isn’t optional: chapter 02’s deep dive explained why all-zero weights would freeze the network in symmetry forever. So: random start, d10 performance, everything still to be earned.

Training needs a single number for “how wrong.” Cross-entropy loss: look up the probability the network gave the true label, take −log. Confident-and-right lands near 0; confident-and-wrong explodes toward infinity — one number, with sharp opinions about overconfidence. And it carries a signature: a fresh 10-class network always starts at loss ≈ 2.3026, no matter whose laptop, whose random seed, whose framework. That’s no coincidence. It is ln 10 — the arithmetic fingerprint of a perfect shrug spread across ten options.

To fix a weight you must know its share of the blame. Backpropagation is the chain rule run in reverse: the error at the output flows backward through the network, layer by layer, and every one of the 101,770 weights receives a gradient — “increase me and the loss rises by exactly this much.” One forward pass, one backward pass, complete blame assignment. The paper that made this the field’s engine — Rumelhart, Hinton & Williams, Nature, October 1986 — runs three pages. Fields have been restarted by less, but not often.

With gradients in hand, the update is an anticlimax: w ← w − 0.01 × gradient. Nudge each of 101,770 knobs a hundredth of a step downhill, on a loss landscape with 101,770 dimensions. That is the whole of gradient descent — the learning rate 0.01 is the step size, and “downhill” is the only instruction ever given. The network never learns “a seven is a bar and a hook.” It only ever follows the local slope; the stroke detectors of chapter 01 are simply what the bottom of this particular valley happens to look like.

Now scale it: all 60,000 digits, ten epochs. A backward pass costs roughly twice a forward one, so the bill reads 101,632 × 3 × 60,000 × 10 ≈ 183 billion multiply-accumulates — paid once, by someone else, before you ever touched the canvas. Loss lands at 0.02; the stroke detectors precipitate; the verdict machinery of chapters 02–03 finally exists. Your two-millisecond answer is the receipt for that mountain: inference is cheap precisely because training was not. The debt column in the counter below fills in at last.

The loose thread from chapter 03, pulled hard. To a dense layer, pixel 407 and pixel 435 — vertical neighbors on your canvas — are just unrelated columns in a spreadsheet. The flattening of chapter 00 destroyed the geometry, and the layer never knew “next to” existed. Shift your seven two pixels and detector #47 finds its violet band empty: confidence 98.3% → 71.0%, and falling with every extra pixel. The failure is not in the weights — they are perfect, for centered digits. It is in the architecture’s worldview: it memorized positions, and your ink moved.

Convolution’s move: shrink the detector to a 5×5 window — 25 weights plus a bias, 26 numbers total — and slide it across the image, reapplying the same kernel at each of 576 positions. The output is a feature map: literally a picture of where the pattern lives. One reusable 26-parameter template replaces the dense layer’s 100,480 position-locked weights, and translation stops being a catastrophe — the detector visits every position, so the top bar is found wherever you happened to draw it. Weight sharing is the whole trick, and it is enough.

LeNet-5 — Yann LeCun’s 1998 blueprint — stacks the idea: 6 kernels sweep the canvas, pooling halves the maps, 16 more kernels read those, pooling again, then small dense layers finish the job: 28×28 → 24×24 → 12×12 → 8×8 → 4×4 → 10 scores. Total: ~61,706 parameters and ~340,908 multiply-accumulates per digit. Sit with that trade: 40% fewer parameters than our dense net, three times the arithmetic — and it reads your shifted seven at 99.1% without blinking. Σ vaults to 442,540.

This is not retro-futurism — LeNet shipped. NCR deployed LeCun’s check-reading system commercially in June 1996, and by 2001 it was reading about 20 million checks a day: roughly one in ten of every check written in the United States, years before “deep learning” was a phrase anyone said out loud. If you mailed a rent check in 1999, odds are decent the amount was read by a convolutional neural network — this same architecture, at production scale, two decades before the hype arrived to explain it.

Scale the recipe and history breaks open. AlexNet (2012): five convolutional layers plus three dense, 60 million parameters, 650,000 neurons, trained for five to six days on two consumer GTX 580 gaming cards. It took ImageNet’s top-5 error to 15.3% while the best non-neural rival managed 26.2% — a 10.8-point cliff that started the modern AI era, embarrassed a decade of hand-engineered vision, and made “just scale it” a research program. Every layer of it is chapter 01’s multiply-add-bend. The lens just got a thousand times bigger, and the world finally noticed.

Rewind the event one last time — further back than chapter 04 went. You never handed the machine a bitmap; you performed a gesture. Pen down, a bar left to right, pen up, a slash to the floor: roughly 60 sampled points, two strokes, in strict order. Rasterizing threw that order away, and both previous readers worked from the corpse. A different family of networks reads the trajectory itself — one point at a time, like a sentence. For this chapter we train one on stroke recordings and hand it your gesture, live.

A recurrent network is one cell, reused. At each timestep it reads the next pen point plus its own previous output — a 128-number hidden state acting as running memory. Point 1: “ink began.” Point 9: “a leftward bar so far.” Point 41: “bar, then a descending slash — this is becoming a seven.” The same weights fire at every step. Unroll the loop and you are looking at a 60-layer network in which every layer is the same layer. Time, handled with a photocopier.

The classic RNN has a fatal leak. Training must send gradients backward through all 60 steps, and every hop multiplies by roughly the same factor. 0.9⁶⁰ ≈ 0.0018. The lesson from the pen-down at step 1 reaches the weights at a five-hundredth of its strength, so the network structurally cannot learn that beginnings matter. Sepp Hochreiter proved this — the vanishing gradient — in his 1991 diploma thesis, in German, years before most of the field could read the warning.

The 1997 fix — Hochreiter & Schmidhuber’s Long Short-Term Memory — gives the cell a protected conveyor belt of memory and three learned gates: forget (what to erase), input (what to write), output (what to reveal). Along the belt, the gradient’s multiplier is exactly 1.0, so lessons survive all 60 steps intact. The cost is real: four gate-sized matrix multiplies per point, 67,072 MACs × 60 ≈ 4 million, and Σ vaults to ≈ 4,468,140 — the most expensive reading of your seven yet. The paper has over 80,000 citations. It earned them.

This cell conquered sequence-land. By September 2016, Google Translate ran on GNMT — eight LSTM layers encoding, eight decoding — cutting translation errors by roughly 60% overnight and chewing through 18 million Chinese→English sentences a day. Then the honest footnote: nine months later, June 2017, Attention Is All You Need introduced the transformer, whose attention reads every timestep at once instead of one by one — and RNNs left the frontier almost as fast as they had taken it. The gesture-reading idea survives. The crown moved.