# Inside SFPU Overflow Bugs: How a 40-Year-Old Rounding Trick Breaks on Modern AI Accelerators

> Source: <https://dev.to/truongsontung/inside-sfpu-overflow-bugs-how-a-40-year-old-rounding-trick-breaks-on-modern-ai-accelerators-n6g>
> Published: 2026-09-09 08:23:54+00:00

`ttnn.softplus(-1e7)` returns `+inf`. Not approximately zero — **infinity**. On a chip that costs thousands of dollars.

``` python
import torch, ttnn
x = torch.tensor([-1e7, -1e8, -1e10], dtype=torch.float32)
t = ttnn.from_torch(x, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)
print(ttnn.softplus(t))  # tensor([inf, inf, nan])
```

The true answer? `softplus(-1e7) = log(1 + exp(-1e7)) ≈ 0`.

The SFPU (Scalar Functional Processing Unit) on Tenstorrent's Blackhole and Wormhole chips computes `exp(x)` for negative `x` using range reduction + Taylor polynomial. The range reduction step needs to round `z = x / ln(2)` to the nearest integer `k`, after which `r = x - k*ln(2)` is the small residual fed into a polynomial.

The rounding trick is from **Hacker's Delight** (Henry S. Warren, Jr., 2003): add the constant `0x4B400000` (= 2^23 + 2^22), reinterpret as int, subtract, and you have a round-to-nearest-integer — but **only if `|z| <= 2^22`**.

```
z + (2^23 + 2^22) is representable in [2^22, 2^23], so the fraction bits
encode the integer part. Outside that range, the bit trick produces garbage.
```

For most activation functions, `z` is naturally bounded. But `softplus_exp_negative` passed `z` **unclamped** to the helper, and for `|x| >= ~8.7e6`, `|z| = |x|/ln(2) > 2^22`, so:

`k_int` instead of a large negative one.`new_exp = p_exp + k_int` becomes large and positive.`new_exp > 0` flush-to-zero guard (meant for underflow) sees a positive exponent and writes it straight into the 8-bit exponent field.`+inf` or `NaN`.

```
// Before:
sfpi::vFloat z = x * INV_LN2;
sfpi::vFloat k = _sfpu_round_to_nearest_int32_(z, k_int);  // 💥 z unbounded

// After:
sfpi::vFloat z = x * INV_LN2;
constexpr float UNDERFLOW_THRESHOLD = -126.5f;
z = sfpi::max(z, UNDERFLOW_THRESHOLD);  // ✅ matches xielu, gelu, etc.
sfpi::vFloat k = _sfpi_round_to_nearest_int32_(z, k_int);
```

The clamp is exact because `exp(x)` underflows to 0 for `x < -126.5` in float32. Clamping `z` to `-126.5` means `k_int ≈ -126`, which gives `new_exp < 0`, so the flush-to-zero guard correctly returns 0 — exactly what `softplus` should return for large negative inputs.

Modern AI accelerators push floating-point to its limits:

`2^-126` to `2^126`
Every other eltwise op in the codebase already clamps — `xielu`, `gelu`, `exp`, `sigmoid` all bound their arguments to the rounding helper. `softplus` was the one that didn't.

The same class of bug appears in `ttnn.reciprocal` (issue #55797): the Blackhole fp32 path uses additive Newton-Raphson refinement (`y = t2*y + y`), which underflows for `|x| >= 2^119`. The multiplicative form (` y = y * (2 - x*y)`) used by `rdiv` and `pow` doesn't have this problem.

The lesson: in subnormal-range arithmetic, **the order of operations matters**. Computing `1 + small` first, then multiplying, preserves precision that `small * large + large` loses.

When you're debugging a chip that costs more than most cars:

And yes — I'm hiring my debugging process as a service. [Contact me on GitHub @truongsontung](https://github.com/truongsontung).