{"slug": "pytorch-backward-is-just-a-graph-traversal-in-disguise", "title": "PyTorch .backward() is just a graph traversal in disguise", "summary": "PyTorch's .backward() method performs a graph traversal that applies the chain rule node-by-node rather than deriving a global formula, according to a technical explainer. The article uses the example z = 2x^2 + 1 with x=3, where the backward pass multiplies local slopes (1, 2, and 6) to yield a gradient of 12 stored in x.grad. This mechanism underpins parameter updates in neural networks and large language models without handwritten derivatives.", "body_md": "# PyTorch .backward() is just a graph traversal in disguise\n\n`requires_grad`\n\nand `.backward()`\n\nat you, plug them into a training loop, and leave you wondering what actually happened under the hood. It feels like magic until you realize that PyTorch is essentially just keeping a meticulous receipt of every operation you perform. If you can do a derivative by hand on a scrap of paper, you've already done the work PyTorch does—you were just slower at it.## The logic of the gradient\n\nBefore hitting the code, remember that a gradient is simply the slope of the ground under your feet. If you're on a hillside in thick fog, the gradient tells you which way is \"down.\" In a model, the horizontal axis is a parameter and the vertical axis is the loss. If the slope is positive, you move left to lower the loss; if it's negative, you move right.\n\nTake $y = x^2$. The derivative is $2x$. If $x = 3$, the slope is $6$. PyTorch handles this without needing the explicit formula:\n\n``` python\nimport torch\n\nx = torch.tensor(3.0, requires_grad=True)\ny = x ** 2\ny.backward()\nprint(x.grad) # tensor(6.)\n```\n\n## The \"Tape\" and the Computation Graph\n\nTensors don't track gradients by default because doing so for every single input would be a massive waste of memory. You opt-in using `requires_grad=True`\n\n. Once you do, PyTorch starts recording.\n\nThis \"recording\" is the computation graph. It's a directed chain where nodes are operations and edges are tensors. When you run a forward pass, PyTorch computes the result and simultaneously builds this graph.\n\n```\nx = torch.tensor(3.0, requires_grad=True)\ny = x ** 2 # node: power\nz = 2 * y + 1 # nodes: multiply, then add\n```\n\nIn this scenario, `x`\n\nis a **leaf** node because it was created directly. Every subsequent result node (like `y`\n\nand `z`\n\n) stores a `grad_fn`\n\n. This isn't the operation itself, but the specific instruction on how to reverse that operation during the backward pass.\n\n```\nprint(z.grad_fn) # <PowBackward0 object at ...>\nprint(y.grad_fn) # <MulBackward0 object at ...>\nprint(x.grad_fn) # None (leaves have no history)\n```\n\n## What actually happens during .backward()\n\nWhen you call `.backward()`\n\n, PyTorch doesn't magically derive a global formula like $dz/dx = 4x$. Instead, it walks the graph from right to left (the backward pass). At each node, it multiplies the incoming gradient by that node's local slope.\n\nFor the equation $z = 2x^2 + 1$ where $x=3$:\n\n1. It starts at $z$ with an implicit gradient of $1$.\n\n2. It hits the `+1`\n\nnode. The slope of a constant addition is $1$. Gradient remains $1$.\n\n3. It hits the `*2`\n\nnode. The slope is $2$. Gradient becomes $1 \\times 2 = 2$.\n\n4. It hits the `**2`\n\nnode. The local slope is $2x$ (which is $6$ when $x=3$). Gradient becomes $2 \\times 6 = 12$.\n\nThe final result stored in `x.grad`\n\nis $12$. This chain-rule traversal is the core of any LLM agent or neural network deployment, allowing the system to update millions of parameters without needing a handwritten derivative for the entire architecture.\n\n[Next Laguna S 2.1 is actually beating Qwen3.5 in my current tests →](/en/threads/6052/)\n\n## All Replies （4）\n\n`detach()`\n\nwas necessary until I actually mapped the graph.`.backward()`\n\ndestroys the graph by default unless you set `retain_graph=True`\n\n.", "url": "https://wpnews.pro/news/pytorch-backward-is-just-a-graph-traversal-in-disguise", "canonical_source": "https://promptcube3.com/en/threads/6087/", "published_at": "2026-08-12 23:05:09+00:00", "updated_at": "2026-08-12 23:19:25.132188+00:00", "lang": "en", "topics": ["machine-learning", "developer-tools"], "entities": ["PyTorch"], "alternates": {"html": "https://wpnews.pro/news/pytorch-backward-is-just-a-graph-traversal-in-disguise", "markdown": "https://wpnews.pro/news/pytorch-backward-is-just-a-graph-traversal-in-disguise.md", "text": "https://wpnews.pro/news/pytorch-backward-is-just-a-graph-traversal-in-disguise.txt", "jsonld": "https://wpnews.pro/news/pytorch-backward-is-just-a-graph-traversal-in-disguise.jsonld"}}