Inside SFPU Overflow Bugs: How a 40-Year-Old Rounding Trick Breaks on Modern AI Accelerators

Tenstorrent's Blackhole and Wormhole AI accelerators return infinity or NaN for large negative inputs to the softplus activation function due to an unclamped range-reduction step in the SFPU (Scalar Functional Processing Unit). The bug, traced to a rounding trick from Hacker's Delight that fails when the argument exceeds 2^22, was fixed by clamping the input to match the underflow threshold, aligning with other elementwise operations. The same class of issue affects ttnn.reciprocal on Blackhole's fp32 path.

ttnn.softplus -1e7 returns +inf . Not approximately zero — infinity . On a chip that costs thousands of dollars. python import torch, ttnn x = torch.tensor -1e7, -1e8, -1e10 , dtype=torch.float32 t = ttnn.from torch x, dtype=ttnn.float32, layout=ttnn.TILE LAYOUT, device=device print ttnn.softplus t tensor inf, inf, nan The true answer? softplus -1e7 = log 1 + exp -1e7 ≈ 0 . The SFPU Scalar Functional Processing Unit on Tenstorrent's Blackhole and Wormhole chips computes exp x for negative x using range reduction + Taylor polynomial. The range reduction step needs to round z = x / ln 2 to the nearest integer k , after which r = x - k ln 2 is the small residual fed into a polynomial. The rounding trick is from Hacker's Delight Henry S. Warren, Jr., 2003 : add the constant 0x4B400000 = 2^23 + 2^22 , reinterpret as int, subtract, and you have a round-to-nearest-integer — but only if |z| <= 2^22 . z + 2^23 + 2^22 is representable in 2^22, 2^23 , so the fraction bits encode the integer part. Outside that range, the bit trick produces garbage. For most activation functions, z is naturally bounded. But softplus exp negative passed z unclamped to the helper, and for |x| = ~8.7e6 , |z| = |x|/ln 2 2^22 , so: k int instead of a large negative one. new exp = p exp + k int becomes large and positive. new exp 0 flush-to-zero guard meant for underflow sees a positive exponent and writes it straight into the 8-bit exponent field. +inf or NaN . // Before: sfpi::vFloat z = x INV LN2; sfpi::vFloat k = sfpu round to nearest int32 z, k int ; // 💥 z unbounded // After: sfpi::vFloat z = x INV LN2; constexpr float UNDERFLOW THRESHOLD = -126.5f; z = sfpi::max z, UNDERFLOW THRESHOLD ; // ✅ matches xielu, gelu, etc. sfpi::vFloat k = sfpi round to nearest int32 z, k int ; The clamp is exact because exp x underflows to 0 for x < -126.5 in float32. Clamping z to -126.5 means k int ≈ -126 , which gives new exp < 0 , so the flush-to-zero guard correctly returns 0 — exactly what softplus should return for large negative inputs. Modern AI accelerators push floating-point to its limits: 2^-126 to 2^126 Every other eltwise op in the codebase already clamps — xielu , gelu , exp , sigmoid all bound their arguments to the rounding helper. softplus was the one that didn't. The same class of bug appears in ttnn.reciprocal issue 55797 : the Blackhole fp32 path uses additive Newton-Raphson refinement y = t2 y + y , which underflows for |x| = 2^119 . The multiplicative form y = y 2 - x y used by rdiv and pow doesn't have this problem. The lesson: in subnormal-range arithmetic, the order of operations matters . Computing 1 + small first, then multiplying, preserves precision that small large + large loses. When you're debugging a chip that costs more than most cars: And yes — I'm hiring my debugging process as a service. Contact me on GitHub @truongsontung https://github.com/truongsontung .