{"slug": "backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees", "title": "Backpropagation by Hand: Two Layers, a Pen, and Then Autograd Agrees", "summary": "A developer manually computed gradients for a two-layer neural network with ReLU activation and squared-error loss, deriving ∂L/∂w1=30, ∂L/∂b1=30, ∂L/∂w2=20, and ∂L/∂b2=10, then verified that PyTorch's autograd produces identical values. The exercise demonstrates how the chain rule propagates gradients backward through a hidden layer, with the loss gradient multiplied by local derivatives at each node.", "body_md": "in the last piece i did autograd, and the whole point of it was that you call `loss.backward()`\n\nonce and pytorch fills in every gradient for you, and a couple of people basically said the same thing back to me, ok that is nice but on a real network with more than one layer what is it actually doing in there. so this time i want to do the thing you will genuinely never do inside a training loop, i want to take a two-layer network and compute every gradient by hand, one node at a time, and then type the same thing into pytorch and watch autograd land on the exact same numbers. the one-neuron version was the warm-up. this is the one where the chain rule has to travel back through a hidden layer, and where the interesting failures live.\n\nTwo layers. one input, a hidden neuron with a ReLU, then an output neuron, then a squared-error loss. that is it. i am going to write it as plain scalars first because the whole trick is visible that way and nothing is hiding inside a matrix.\n\n```\nx ──▶ h1 = w1·x + b1 ──▶ a1 = ReLU(h1) ──▶ h2 = w2·a1 + b2 ──▶ L = (h2 − y)²\n```\n\nfour parameters to find gradients for: `w1`\n\n, `b1`\n\n, `w2`\n\n, `b2`\n\n. and i am picking numbers so there is nothing to hide behind:\n\n```\nx  = 1.0\nw1 = 2.0    b1 = 0.0\nw2 = 3.0    b2 = 1.0\ny  = 2.0     (the target)\n```\n\nyou compute the answer first, and you keep every intermediate value, because the backward pass needs them. this is not optional bookkeeping, the local derivatives literally reuse these numbers.\n\n```\nh1 = w1·x + b1 = 2·1 + 0   = 2.0\na1 = ReLU(h1)  = max(0, 2)  = 2.0\nh2 = w2·a1 + b2 = 3·2 + 1   = 7.0\nL  = (h2 − y)²  = (7 − 2)²   = 25.0\n```\n\nloss is 25. now the only question that matters is how each of the four parameters should move to make that 25 smaller, and that is exactly what a gradient is, `∂L/∂w1`\n\nand so on, the slope of the\n\nloss with respect to that one number while everything else is held still.\n\nthe entire method is this, and it does not get more complicated no matter how deep the net is: at each node you take the gradient arriving from the right, and you multiply it by that node's own\n\nlocal derivative, and you pass the result further left. that is the chain rule and there is nothing else to it.\n\n**loss → h2.** `L = (h2 − y)²`\n\n, so the derivative of the loss with respect to `h2`\n\nis `2(h2 − y)`\n\n.\n\n```\n∂L/∂h2 = 2·(7 − 2) = 10.0\n```\n\n**h2 → the output parameters.** `h2 = w2·a1 + b2`\n\n. the derivative of `h2`\n\nwith respect to `w2`\n\nis just `a1`\n\n, and with respect to `b2`\n\nit is 1, so multiply each by the 10 coming in:\n\n```\n∂L/∂w2 = 10 · a1  = 10 · 2 = 20.0\n∂L/∂b2 = 10 · 1   = 10.0\n```\n\nso far identical in spirit to the one-neuron case. here is the part that only shows up once you have a hidden layer.\n\n**h2 → a1, i.e. keep going left into the first layer.** `h2 = w2·a1 + b2`\n\n, so the derivative of `h2`\n\nwith respect to `a1`\n\nis `w2`\n\n. the gradient does not stop at the output layer, it flows backthrough `w2`\n\ninto the hidden activation:\n\n```\n∂L/∂a1 = 10 · w2 = 10 · 3 = 30.0\n```\n\n**a1 → h1, through the ReLU.** ReLU passes its input when it was positive and its derivative is 1 there, 0 when the input was negative. our `h1`\n\nwas 2, positive, so the gate is open:\n\n```\n∂a1/∂h1 = 1     (because h1 > 0)\n∂L/∂h1  = 30 · 1 = 30.0\n```\n\n**h1 → the first-layer parameters.** `h1 = w1·x + b1`\n\n, derivative with respect to `w1`\n\nis `x`\n\n, with respect to `b1`\n\nis 1:\n\n```\n∂L/∂w1 = 30 · x = 30 · 1 = 30.0\n∂L/∂b1 = 30 · 1 = 30.0\n```\n\ndone, by hand, four gradients:\n\n```\n∂L/∂w1 = 30    ∂L/∂b1 = 30    ∂L/∂w2 = 20    ∂L/∂b2 = 10\n```\n\nnotice the shape of what happened. the `10`\n\ncomputed at the output got carried all the way back to the first layer, multiplied by `w2`\n\non the way through, then by the ReLU's `1`\n\n, then by `x`\n\n. every\n\ngradient in the network is that same number from the loss, multiplied by a chain of local slopes between it and the parameter. deeper nets are just longer chains.\n\nsame numbers, same five-ish lines, mark the four parameters as needing gradients, run forward, call backward once, print:\n\n``` python\nimport torch\n\nx  = torch.tensor(1.0)\nw1 = torch.tensor(2.0, requires_grad=True)\nb1 = torch.tensor(0.0, requires_grad=True)\nw2 = torch.tensor(3.0, requires_grad=True)\nb2 = torch.tensor(1.0, requires_grad=True)\ny  = torch.tensor(2.0)\n\nh1 = w1 * x + b1\na1 = torch.relu(h1)\nh2 = w2 * a1 + b2\nloss = (h2 - y) ** 2\n\nloss.backward()\n\nprint(f\"loss   = {loss.item()}\")     # 25.0\nprint(f\"dL/dw1 = {w1.grad.item()}\")  # 30.0\nprint(f\"dL/db1 = {b1.grad.item()}\")  # 30.0\nprint(f\"dL/dw2 = {w2.grad.item()}\")  # 20.0\nprint(f\"dL/db2 = {b2.grad.item()}\")  # 10.0\n```\n\nall four match, and the loss matches. autograd traced the same chain of operations, stored the same intermediate values on the forward pass, and multiplied the same local derivatives on the way back.\n\nthe only thing it did that you did not is bookkeeping, it just did it without asking you and it would do it the same way with ten million parameters instead of four.\n\nif you read nothing else in this article, run that block and put it next to the hand calculation above. the whole point is the moment where the pen and the computer agree.\n\nkeep the network the same but break the hidden neuron on purpose. flip `w1`\n\nto `-2`\n\nand leave everything else. now the first layer produces a negative pre-activation:\n\n```\nh1 = w1·x + b1 = -2·1 + 0 = -2.0\na1 = ReLU(-2)  = 0.0\nh2 = w2·a1 + b2 = 3·0 + 1 = 1.0\nL  = (1 − 2)²   = 1.0\n```\n\nrun backward on this one and look at what comes out:\n\n```\nw1 = torch.tensor(-2.0, requires_grad=True)   # the only change\nb1 = torch.tensor(0.0,  requires_grad=True)\nw2 = torch.tensor(3.0,  requires_grad=True)\nb2 = torch.tensor(1.0,  requires_grad=True)\nx, y = torch.tensor(1.0), torch.tensor(2.0)\n\nh1 = w1 * x + b1\na1 = torch.relu(h1)\nh2 = w2 * a1 + b2\nloss = (h2 - y) ** 2\nloss.backward()\n\nprint(w1.grad.item(), b1.grad.item())   # 0.0 0.0\nprint(w2.grad.item(), b2.grad.item())   # 0.0 -2.0\n```\n\nthree of the four gradients are zero, and the loss is not, the network is wrong and mostly getting no signal about it. and because you just did the hand version you can say exactly why each zero\n\nhappened instead of guessing.\n\n`w1`\n\nand `b1`\n\nare zero because the ReLU's derivative is 0 when its input was negative, and that 0 sits in the middle of the chain, so everything behind it gets multiplied down to nothing. that is\n\nthe dead-ReLU problem and this is it in miniature, a neuron whose input is negative gets no gradient and stops learning, quietly, no error printed.\n\nbut look at `w2`\n\n, it is also zero, and the ReLU is not the reason. `∂L/∂w2 = ∂L/∂h2 · a1`\n\n, and `a1`\n\nis 0 here, so the weight that reads *from* the dead neuron gets a zero gradient too, because you\n\ncannot learn how to weight an input that is always zero. so a single dead neuron in the hidden layer takes out both the parameters feeding it and the weight reading out of it.\n\n`b2`\n\nis the survivor, `∂L/∂b2 = ∂L/∂h2 = -2`\n\n, the output bias sits after the dead neuron so the zero never reaches it, and it keeps learning. that is the whole reason this is worth doing on two\n\nlayers and not one, on one neuron you just see \"gradient is zero\", on two you see the zero spread along the chain and stop where the chain does.\n\nthis is the payoff of the exercise. you will never backprop by hand in a real loop, autograd owns that. but the day a model just sits there and does not improve and you print a gradient and it is 0,\n\nyou will not be staring at it, you will go straight to asking what on the path back is multiplying by zero, a dead ReLU, a `detach`\n\nyou forgot about, an activation that saturated, and you will know that\n\na zero at one node zeros everything behind it but not in front of it, because you have already watched it happen on four parameters you could hold in your head.\n\n**the gradient does not stop at the layer that produced the loss.** the most common beginner mental model is that each layer computes its own gradient locally and independently. it does not. the `10`\n\nfrom the output travelled all the way to `w1`\n\n. every layer's gradient depends on every layer after it, that dependence is the `· w2`\n\nstep, and it is why the order is strictly right to left and why you cannot parallelise the backward pass across depth the way you can the forward one.\n\n**gradients accumulate, they do not overwrite.** if you run `loss.backward()`\n\ntwice without clearing,\n\nthe second set of numbers gets added onto the first and you will read `60, 60, 40, 20`\n\non the second call and wonder what happened. that is deliberate, it is what lets you sum gradients over several mini-batches, but it means `optimizer.zero_grad()`\n\nbefore every backward is load-bearing, not boilerplate. i wrote about why that is in the autograd piece.\n\ntake the healthy network again, `x=1, w1=2, b1=0, w2=3, b2=1, y=2`\n\n, and this time add a second hidden step of your own, say multiply `a1`\n\nby another weight `w3 = 0.5`\n\nbefore the output. write out\n\nthe full chain for `∂L/∂w1`\n\non paper, there is just one more factor in it now, then check against autograd. it takes a few minutes and it is worth more than rereading this.\n\nthen the mean one, set `b1 = -5`\n\nso the hidden ReLU dies, predict which of the four gradients go to zero *before* you run it, and see if you called it right. if you can predict the zeros you have\n\nactually got the chain rule, if you cannot you have more to gain from running it than from any amount of reading.\n\n**what is the worst \"model just would not learn and the gradient was zero\" you have hit?** for me the first real one was a dead ReLU exactly like this, a whole layer of them, and it cost me most of a day before i thought to print a gradient. i am curious whether that is the usual first one or whether everyone finds their own way to multiply by zero.\n\n*This is one chapter's worth of an idea from my book,* **PyTorch From Ground Up**, *which builds everything from tensors upward so nothing stays vague. If it helped:\n8 chapters are free, no email required, there's a free one-page tensor cheat-sheet here, every example runs in\nthe companion notebooks on GitHub, and the full book is on Leanpub or in\npaperback and Kindle on Amazon.*\n\n**How Training Actually Works**, the part where the training loop stops being magic:\n\nThe shape mechanics underneath all of it, worth having solid first:\n\nComing next in *How Training Actually Works*:", "url": "https://wpnews.pro/news/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees", "canonical_source": "https://dev.to/pytorchfromgroundup/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees-13i6", "published_at": "2026-08-23 07:20:10+00:00", "updated_at": "2026-08-23 07:43:10.877036+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "developer-tools"], "entities": ["PyTorch"], "alternates": {"html": "https://wpnews.pro/news/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees", "markdown": "https://wpnews.pro/news/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees.md", "text": "https://wpnews.pro/news/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees.txt", "jsonld": "https://wpnews.pro/news/backpropagation-by-hand-two-layers-a-pen-and-then-autograd-agrees.jsonld"}}