{"slug": "pytorch-keepdim-stop-the-silent-broadcasting-bugs", "title": "PyTorch keepdim: Stop the silent broadcasting bugs", "summary": "PyTorch's keepdim parameter prevents silent broadcasting bugs when reducing tensors, according to a practical tutorial. Forgetting keepdim can cause either a RuntimeError with non-square tensors or mathematically wrong results with square tensors, as the dimension alignment silently shifts. The tutorial advises always defaulting to keepdim=True for reductions in AI workflows.", "body_md": "# PyTorch keepdim: Stop the silent broadcasting bugs\n\n`(2, 3)`\n\ntensor reduced over `dim=1`\n\nbecomes `(2,)`\n\nby default, but `(2, 1)`\n\nif you set `keepdim=True`\n\n. The values are identical, but that single `1`\n\nin the shape is the difference between a successful normalization and a cryptic `RuntimeError`\n\n—or worse, a calculation that runs but produces mathematically wrong results.## The mechanics of reduction\n\nIn PyTorch, when you use `sum`\n\n, `mean`\n\n, `max`\n\n, or `min`\n\n, the dimension you specify is the one that vanishes.\n\n``` python\nimport torch\n\nm = torch.tensor([[1., 2., 3.],\n [4., 5., 6.]]) # shape (2, 3)\n\nprint(m.sum(dim=0).shape) # torch.Size([3]) - rows collapse, columns remain\nprint(m.sum(dim=1).shape) # torch.Size([2]) - columns collapse, rows remain\n```\n\nIf you name a dimension, it's gone. `keepdim=True`\n\nsimply prevents this deletion by leaving a length-1 placeholder.\n\n```\nrow_sum = m.sum(dim=1) # shape (2,)\nrow_sum_k = m.sum(dim=1, keepdim=True) # shape (2, 1)\n\nprint(row_sum) \n# tensor([ 6., 15.])\nprint(row_sum_k) \n# tensor([[ 6.],\n# [15.]])\n```\n\n## Why this is a practical tutorial for broadcasting\n\nMost people use reductions to perform operations back on the original tensor—like row-wise normalization or subtracting a mean. This requires broadcasting. PyTorch aligns shapes from the right; it treats a `1`\n\nas \"stretch me to fit.\"\n\nA `(2, 1)`\n\ntensor broadcasts perfectly against a `(2, 3)`\n\ntensor because the `1`\n\nexpands to `3`\n\n. A `(2,)`\n\ntensor does not align.\n\n```\n# Correct: keepdim=True allows (2, 1) to broadcast against (2, 3)\nnormed = m / m.sum(dim=1, keepdim=True)\nprint(normed.sum(dim=1)) # tensor([1., 1.]) - Works perfectly\n```\n\n## The \"Silent Bug\" Trap\n\nThe danger is that forgetting `keepdim`\n\nfails in two different ways depending on your tensor shape.\n\n**Scenario A: The Loud Failure (Non-Square Tensors)**\n\nIf you have a `(2, 3)`\n\ntensor and divide by a `(2,)`\n\nsum, PyTorch tries to align the `3`\n\nwith the `2`\n\n. It crashes immediately with a `RuntimeError`\n\n. This is actually the best-case scenario because you know exactly where the bug is.\n\n**Scenario B: The Silent Failure (Square Tensors)**\n\nIf your tensor is `(3, 3)`\n\n, the row sum is `(3,)`\n\n. PyTorch aligns the rightmost dimensions (3 and 3), and it *works*—but it does the wrong thing. Instead of dividing each row by its own sum, it divides each column by the row sums. You get a result, but your data is now garbage.\n\nIf you're building a real-world AI workflow or a custom LLM agent layer, always default to `keepdim=True`\n\nfor reductions. It removes the ambiguity and prevents these shape-shifting bugs from leaking into your gradients.\n\n[Next Why I stopped trusting AI agents to grade their own work →](/en/threads/2731/)", "url": "https://wpnews.pro/news/pytorch-keepdim-stop-the-silent-broadcasting-bugs", "canonical_source": "https://promptcube3.com/en/threads/2745/", "published_at": "2026-07-24 13:46:18+00:00", "updated_at": "2026-07-24 14:08:32.558283+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["PyTorch"], "alternates": {"html": "https://wpnews.pro/news/pytorch-keepdim-stop-the-silent-broadcasting-bugs", "markdown": "https://wpnews.pro/news/pytorch-keepdim-stop-the-silent-broadcasting-bugs.md", "text": "https://wpnews.pro/news/pytorch-keepdim-stop-the-silent-broadcasting-bugs.txt", "jsonld": "https://wpnews.pro/news/pytorch-keepdim-stop-the-silent-broadcasting-bugs.jsonld"}}