PyTorch .backward() is just a graph traversal in disguise

PyTorch's .backward() method performs a graph traversal that applies the chain rule node-by-node rather than deriving a global formula, according to a technical explainer. The article uses the example z = 2x^2 + 1 with x=3, where the backward pass multiplies local slopes (1, 2, and 6) to yield a gradient of 12 stored in x.grad. This mechanism underpins parameter updates in neural networks and large language models without handwritten derivatives.

PyTorch .backward is just a graph traversal in disguise requires grad and .backward at you, plug them into a training loop, and leave you wondering what actually happened under the hood. It feels like magic until you realize that PyTorch is essentially just keeping a meticulous receipt of every operation you perform. If you can do a derivative by hand on a scrap of paper, you've already done the work PyTorch does—you were just slower at it. The logic of the gradient Before hitting the code, remember that a gradient is simply the slope of the ground under your feet. If you're on a hillside in thick fog, the gradient tells you which way is "down." In a model, the horizontal axis is a parameter and the vertical axis is the loss. If the slope is positive, you move left to lower the loss; if it's negative, you move right. Take $y = x^2$. The derivative is $2x$. If $x = 3$, the slope is $6$. PyTorch handles this without needing the explicit formula: python import torch x = torch.tensor 3.0, requires grad=True y = x 2 y.backward print x.grad tensor 6. The "Tape" and the Computation Graph Tensors don't track gradients by default because doing so for every single input would be a massive waste of memory. You opt-in using requires grad=True . Once you do, PyTorch starts recording. This "recording" is the computation graph. It's a directed chain where nodes are operations and edges are tensors. When you run a forward pass, PyTorch computes the result and simultaneously builds this graph. x = torch.tensor 3.0, requires grad=True y = x 2 node: power z = 2 y + 1 nodes: multiply, then add In this scenario, x is a leaf node because it was created directly. Every subsequent result node like y and z stores a grad fn . This isn't the operation itself, but the specific instruction on how to reverse that operation during the backward pass. print z.grad fn <PowBackward0 object at ... print y.grad fn <MulBackward0 object at ... print x.grad fn None leaves have no history What actually happens during .backward When you call .backward , PyTorch doesn't magically derive a global formula like $dz/dx = 4x$. Instead, it walks the graph from right to left the backward pass . At each node, it multiplies the incoming gradient by that node's local slope. For the equation $z = 2x^2 + 1$ where $x=3$: 1. It starts at $z$ with an implicit gradient of $1$. 2. It hits the +1 node. The slope of a constant addition is $1$. Gradient remains $1$. 3. It hits the 2 node. The slope is $2$. Gradient becomes $1 \times 2 = 2$. 4. It hits the 2 node. The local slope is $2x$ which is $6$ when $x=3$ . Gradient becomes $2 \times 6 = 12$. The final result stored in x.grad is $12$. This chain-rule traversal is the core of any LLM agent or neural network deployment, allowing the system to update millions of parameters without needing a handwritten derivative for the entire architecture. Next Laguna S 2.1 is actually beating Qwen3.5 in my current tests → /en/threads/6052/ All Replies （4） detach was necessary until I actually mapped the graph. .backward destroys the graph by default unless you set retain graph=True .