How One Small Push Doth Topple the Whole Procession

Welcome back

Last time, we played detective. When the network makes a mistake, backpropagation follows the trail of blame backward, through every layer, to find which weights were most responsible. Today we’ll look at the quiet rule that makes that trail possible. It’s called the chain rule, and the best way to see it is with a long line of dominoes.

A line of falling tiles

Picture a row of dominoes standing on a kitchen table. You nudge the first one. It tips into the second, the second tips into the third, and so on, down the line, until the last one clicks onto the table.

Now ask a simple question: if I push the first domino a little harder, how much harder does the last one fall?

You can’t answer that by looking at the first and last dominoes alone. You have to think about each handoff along the way. How much does the first domino pass to the second? How much does the second pass to the third? Each tile takes the push it was given and hands on some of it, maybe a bit more, maybe a bit less.

The total effect, from the first push to the last fall, is all those little handoffs multiplied together.

The chain rule, in plain words

A neural network is a long line of dominoes too. A weight deep inside the network affects one neuron. That neuron affects the next layer. That layer affects the one after it. Eventually, it all reaches the output, and the output decides the loss, our measure of how wrong the network was.

Backpropagation needs to know: if I nudge this one weight, how much does the loss change? That’s the question from the kitchen table, just asked about a network.

The chain rule gives the answer. It says:

> The effect of a change at the start of a chain equals the effect of each link, multiplied together, all the way down the line.

So if a weight’s nudge makes its neuron change by a factor of two, and that neuron changes the next layer by a factor of a half, and that layer changes the loss by a factor of three, then the weight’s total effect on the loss is two times a half times three. That works out to three.

No single link has to understand the whole chain. Each one only needs to know how it passes things on to its neighbour.

Why this makes learning possible

This is the quiet genius of backpropagation. Instead of calculating every weight’s effect from scratch, the network starts at the end, where the loss is, and works backward one layer at a time. At each layer it takes the “effect so far” and multiplies in that layer’s own small handoff. Then it passes the result back to the layer before.

It’s like walking back along the line of dominoes, noting at each tile, “This one passed on about this much.” By the time you reach the start, you know exactly how much the first push mattered.

Because of this, a network with millions of weights can work out how to adjust every single one of them in roughly the time it takes to run forward once. Without the chain rule, deep networks would simply be too slow to train.

What to carry with you