# Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees > Source: > Published: 2026-08-23 07:20:10+00:00 in the last piece i did autograd, and the whole point of it was that you call `loss.backward()` once and pytorch fills in every gradient for you, and a couple of people basically said the same thing back to me, ok that is nice but on a real network with more than one layer what is it actually doing in there. so this time i want to do the thing you will genuinely never do inside a training loop, i want to take a two-layer network and compute every gradient by hand, one node at a time, and then type the same thing into pytorch and watch autograd land on the exact same numbers. the one-neuron version was the warm-up. this is the one where the chain rule has to travel back through a hidden layer, and where the interesting failures live. Two layers. one input, a hidden neuron with a ReLU, then an output neuron, then a squared-error loss. that is it. i am going to write it as plain scalars first because the whole trick is visible that way and nothing is hiding inside a matrix. ``` x ──▶ h1 = w1·x + b1 ──▶ a1 = ReLU(h1) ──▶ h2 = w2·a1 + b2 ──▶ L = (h2 − y)² ``` four parameters to find gradients for: `w1` , `b1` , `w2` , `b2` . and i am picking numbers so there is nothing to hide behind: ``` x = 1.0 w1 = 2.0 b1 = 0.0 w2 = 3.0 b2 = 1.0 y = 2.0 (the target) ``` you compute the answer first, and you keep every intermediate value, because the backward pass needs them. this is not optional bookkeeping, the local derivatives literally reuse these numbers. ``` h1 = w1·x + b1 = 2·1 + 0 = 2.0 a1 = ReLU(h1) = max(0, 2) = 2.0 h2 = w2·a1 + b2 = 3·2 + 1 = 7.0 L = (h2 − y)² = (7 − 2)² = 25.0 ``` loss is 25. now the only question that matters is how each of the four parameters should move to make that 25 smaller, and that is exactly what a gradient is, `∂L/∂w1` and so on, the slope of the loss with respect to that one number while everything else is held still. the entire method is this, and it does not get more complicated no matter how deep the net is: at each node you take the gradient arriving from the right, and you multiply it by that node's own local derivative, and you pass the result further left. that is the chain rule and there is nothing else to it. **loss → h2.** `L = (h2 − y)²` , so the derivative of the loss with respect to `h2` is `2(h2 − y)` . ``` ∂L/∂h2 = 2·(7 − 2) = 10.0 ``` **h2 → the output parameters.** `h2 = w2·a1 + b2` . the derivative of `h2` with respect to `w2` is just `a1` , and with respect to `b2` it is 1, so multiply each by the 10 coming in: ``` ∂L/∂w2 = 10 · a1 = 10 · 2 = 20.0 ∂L/∂b2 = 10 · 1 = 10.0 ``` so far identical in spirit to the one-neuron case. here is the part that only shows up once you have a hidden layer. **h2 → a1, i.e. keep going left into the first layer.** `h2 = w2·a1 + b2` , so the derivative of `h2` with respect to `a1` is `w2` . the gradient does not stop at the output layer, it flows backthrough `w2` into the hidden activation: ``` ∂L/∂a1 = 10 · w2 = 10 · 3 = 30.0 ``` **a1 → h1, through the ReLU.** ReLU passes its input when it was positive and its derivative is 1 there, 0 when the input was negative. our `h1` was 2, positive, so the gate is open: ``` ∂a1/∂h1 = 1 (because h1 > 0) ∂L/∂h1 = 30 · 1 = 30.0 ``` **h1 → the first-layer parameters.** `h1 = w1·x + b1` , derivative with respect to `w1` is `x` , with respect to `b1` is 1: ``` ∂L/∂w1 = 30 · x = 30 · 1 = 30.0 ∂L/∂b1 = 30 · 1 = 30.0 ``` done, by hand, four gradients: ``` ∂L/∂w1 = 30 ∂L/∂b1 = 30 ∂L/∂w2 = 20 ∂L/∂b2 = 10 ``` notice the shape of what happened. the `10` computed at the output got carried all the way back to the first layer, multiplied by `w2` on the way through, then by the ReLU's `1` , then by `x` . every gradient in the network is that same number from the loss, multiplied by a chain of local slopes between it and the parameter. deeper nets are just longer chains. same numbers, same five-ish lines, mark the four parameters as needing gradients, run forward, call backward once, print: ``` python import torch x = torch.tensor(1.0) w1 = torch.tensor(2.0, requires_grad=True) b1 = torch.tensor(0.0, requires_grad=True) w2 = torch.tensor(3.0, requires_grad=True) b2 = torch.tensor(1.0, requires_grad=True) y = torch.tensor(2.0) h1 = w1 * x + b1 a1 = torch.relu(h1) h2 = w2 * a1 + b2 loss = (h2 - y) ** 2 loss.backward() print(f"loss = {loss.item()}") # 25.0 print(f"dL/dw1 = {w1.grad.item()}") # 30.0 print(f"dL/db1 = {b1.grad.item()}") # 30.0 print(f"dL/dw2 = {w2.grad.item()}") # 20.0 print(f"dL/db2 = {b2.grad.item()}") # 10.0 ``` all four match, and the loss matches. autograd traced the same chain of operations, stored the same intermediate values on the forward pass, and multiplied the same local derivatives on the way back. the only thing it did that you did not is bookkeeping, it just did it without asking you and it would do it the same way with ten million parameters instead of four. if you read nothing else in this article, run that block and put it next to the hand calculation above. the whole point is the moment where the pen and the computer agree. keep the network the same but break the hidden neuron on purpose. flip `w1` to `-2` and leave everything else. now the first layer produces a negative pre-activation: ``` h1 = w1·x + b1 = -2·1 + 0 = -2.0 a1 = ReLU(-2) = 0.0 h2 = w2·a1 + b2 = 3·0 + 1 = 1.0 L = (1 − 2)² = 1.0 ``` run backward on this one and look at what comes out: ``` w1 = torch.tensor(-2.0, requires_grad=True) # the only change b1 = torch.tensor(0.0, requires_grad=True) w2 = torch.tensor(3.0, requires_grad=True) b2 = torch.tensor(1.0, requires_grad=True) x, y = torch.tensor(1.0), torch.tensor(2.0) h1 = w1 * x + b1 a1 = torch.relu(h1) h2 = w2 * a1 + b2 loss = (h2 - y) ** 2 loss.backward() print(w1.grad.item(), b1.grad.item()) # 0.0 0.0 print(w2.grad.item(), b2.grad.item()) # 0.0 -2.0 ``` three of the four gradients are zero, and the loss is not, the network is wrong and mostly getting no signal about it. and because you just did the hand version you can say exactly why each zero happened instead of guessing. `w1` and `b1` are zero because the ReLU's derivative is 0 when its input was negative, and that 0 sits in the middle of the chain, so everything behind it gets multiplied down to nothing. that is the dead-ReLU problem and this is it in miniature, a neuron whose input is negative gets no gradient and stops learning, quietly, no error printed. but look at `w2` , it is also zero, and the ReLU is not the reason. `∂L/∂w2 = ∂L/∂h2 · a1` , and `a1` is 0 here, so the weight that reads *from* the dead neuron gets a zero gradient too, because you cannot learn how to weight an input that is always zero. so a single dead neuron in the hidden layer takes out both the parameters feeding it and the weight reading out of it. `b2` is the survivor, `∂L/∂b2 = ∂L/∂h2 = -2` , the output bias sits after the dead neuron so the zero never reaches it, and it keeps learning. that is the whole reason this is worth doing on two layers and not one, on one neuron you just see "gradient is zero", on two you see the zero spread along the chain and stop where the chain does. this is the payoff of the exercise. you will never backprop by hand in a real loop, autograd owns that. but the day a model just sits there and does not improve and you print a gradient and it is 0, you will not be staring at it, you will go straight to asking what on the path back is multiplying by zero, a dead ReLU, a `detach` you forgot about, an activation that saturated, and you will know that a zero at one node zeros everything behind it but not in front of it, because you have already watched it happen on four parameters you could hold in your head. **the gradient does not stop at the layer that produced the loss.** the most common beginner mental model is that each layer computes its own gradient locally and independently. it does not. the `10` from the output travelled all the way to `w1` . every layer's gradient depends on every layer after it, that dependence is the `· w2` step, and it is why the order is strictly right to left and why you cannot parallelise the backward pass across depth the way you can the forward one. **gradients accumulate, they do not overwrite.** if you run `loss.backward()` twice without clearing, the second set of numbers gets added onto the first and you will read `60, 60, 40, 20` on the second call and wonder what happened. that is deliberate, it is what lets you sum gradients over several mini-batches, but it means `optimizer.zero_grad()` before every backward is load-bearing, not boilerplate. i wrote about why that is in the autograd piece. take the healthy network again, `x=1, w1=2, b1=0, w2=3, b2=1, y=2` , and this time add a second hidden step of your own, say multiply `a1` by another weight `w3 = 0.5` before the output. write out the full chain for `∂L/∂w1` on paper, there is just one more factor in it now, then check against autograd. it takes a few minutes and it is worth more than rereading this. then the mean one, set `b1 = -5` so the hidden ReLU dies, predict which of the four gradients go to zero *before* you run it, and see if you called it right. if you can predict the zeros you have actually got the chain rule, if you cannot you have more to gain from running it than from any amount of reading. **what is the worst "model just would not learn and the gradient was zero" you have hit?** for me the first real one was a dead ReLU exactly like this, a whole layer of them, and it cost me most of a day before i thought to print a gradient. i am curious whether that is the usual first one or whether everyone finds their own way to multiply by zero. *This is one chapter's worth of an idea from my book,* **PyTorch From Ground Up**, *which builds everything from tensors upward so nothing stays vague. If it helped: 8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in the companion notebooks on GitHub, and the full book is on Leanpub or in paperback and Kindle on Amazon.* **How Training Actually Works**, the part where the training loop stops being magic: The shape mechanics underneath all of it, worth having solid first: Coming next in *How Training Actually Works*: