{"slug": "inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai", "title": "Inside SFPU Overflow Bugs: How a 40-Year-Old Rounding Trick Breaks on Modern AI Accelerators", "summary": "Tenstorrent's Blackhole and Wormhole AI accelerators return infinity or NaN for large negative inputs to the softplus activation function due to an unclamped range-reduction step in the SFPU (Scalar Functional Processing Unit). The bug, traced to a rounding trick from Hacker's Delight that fails when the argument exceeds 2^22, was fixed by clamping the input to match the underflow threshold, aligning with other elementwise operations. The same class of issue affects ttnn.reciprocal on Blackhole's fp32 path.", "body_md": "`ttnn.softplus(-1e7)` returns `+inf`. Not approximately zero — **infinity**. On a chip that costs thousands of dollars.\n\n``` python\nimport torch, ttnn\nx = torch.tensor([-1e7, -1e8, -1e10], dtype=torch.float32)\nt = ttnn.from_torch(x, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)\nprint(ttnn.softplus(t))  # tensor([inf, inf, nan])\n```\n\nThe true answer? `softplus(-1e7) = log(1 + exp(-1e7)) ≈ 0`.\n\nThe SFPU (Scalar Functional Processing Unit) on Tenstorrent's Blackhole and Wormhole chips computes `exp(x)` for negative `x` using range reduction + Taylor polynomial. The range reduction step needs to round `z = x / ln(2)` to the nearest integer `k`, after which `r = x - k*ln(2)` is the small residual fed into a polynomial.\n\nThe rounding trick is from **Hacker's Delight** (Henry S. Warren, Jr., 2003): add the constant `0x4B400000` (= 2^23 + 2^22), reinterpret as int, subtract, and you have a round-to-nearest-integer — but **only if `|z| <= 2^22`**.\n\n```\nz + (2^23 + 2^22) is representable in [2^22, 2^23], so the fraction bits\nencode the integer part. Outside that range, the bit trick produces garbage.\n```\n\nFor most activation functions, `z` is naturally bounded. But `softplus_exp_negative` passed `z` **unclamped** to the helper, and for `|x| >= ~8.7e6`, `|z| = |x|/ln(2) > 2^22`, so:\n\n`k_int` instead of a large negative one.`new_exp = p_exp + k_int` becomes large and positive.`new_exp > 0` flush-to-zero guard (meant for underflow) sees a positive exponent and writes it straight into the 8-bit exponent field.`+inf` or `NaN`.\n\n```\n// Before:\nsfpi::vFloat z = x * INV_LN2;\nsfpi::vFloat k = _sfpu_round_to_nearest_int32_(z, k_int);  // 💥 z unbounded\n\n// After:\nsfpi::vFloat z = x * INV_LN2;\nconstexpr float UNDERFLOW_THRESHOLD = -126.5f;\nz = sfpi::max(z, UNDERFLOW_THRESHOLD);  // ✅ matches xielu, gelu, etc.\nsfpi::vFloat k = _sfpi_round_to_nearest_int32_(z, k_int);\n```\n\nThe clamp is exact because `exp(x)` underflows to 0 for `x < -126.5` in float32. Clamping `z` to `-126.5` means `k_int ≈ -126`, which gives `new_exp < 0`, so the flush-to-zero guard correctly returns 0 — exactly what `softplus` should return for large negative inputs.\n\nModern AI accelerators push floating-point to its limits:\n\n`2^-126` to `2^126`\nEvery other eltwise op in the codebase already clamps — `xielu`, `gelu`, `exp`, `sigmoid` all bound their arguments to the rounding helper. `softplus` was the one that didn't.\n\nThe same class of bug appears in `ttnn.reciprocal` (issue #55797): the Blackhole fp32 path uses additive Newton-Raphson refinement (`y = t2*y + y`), which underflows for `|x| >= 2^119`. The multiplicative form (` y = y * (2 - x*y)`) used by `rdiv` and `pow` doesn't have this problem.\n\nThe lesson: in subnormal-range arithmetic, **the order of operations matters**. Computing `1 + small` first, then multiplying, preserves precision that `small * large + large` loses.\n\nWhen you're debugging a chip that costs more than most cars:\n\nAnd yes — I'm hiring my debugging process as a service. [Contact me on GitHub @truongsontung](https://github.com/truongsontung).", "url": "https://wpnews.pro/news/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai", "canonical_source": "https://dev.to/truongsontung/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai-accelerators-n6g", "published_at": "2026-09-09 08:23:54+00:00", "updated_at": "2026-09-09 08:30:54.325377+00:00", "lang": "en", "topics": ["ai-chips", "ai-infrastructure", "developer-tools"], "entities": ["Tenstorrent", "Blackhole", "Wormhole", "SFPU", "ttnn", "Hacker's Delight", "Henry S. Warren, Jr."], "alternates": {"html": "https://wpnews.pro/news/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai", "markdown": "https://wpnews.pro/news/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai.md", "text": "https://wpnews.pro/news/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai.txt", "jsonld": "https://wpnews.pro/news/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai.jsonld"}}