ttnn.softplus(-1e7) returns +inf. Not approximately zero — infinity. On a chip that costs thousands of dollars.
import torch, ttnn
x = torch.tensor([-1e7, -1e8, -1e10], dtype=torch.float32)
t = ttnn.from_torch(x, dtype=ttnn.float32, layout=ttnn.TILE_LAYOUT, device=device)
print(ttnn.softplus(t)) # tensor([inf, inf, nan])
The true answer? softplus(-1e7) = log(1 + exp(-1e7)) ≈ 0.
The SFPU (Scalar Functional Processing Unit) on Tenstorrent's Blackhole and Wormhole chips computes exp(x) for negative x using range reduction + Taylor polynomial. The range reduction step needs to round z = x / ln(2) to the nearest integer k, after which r = x - k*ln(2) is the small residual fed into a polynomial.
The rounding trick is from Hacker's Delight (Henry S. Warren, Jr., 2003): add the constant 0x4B400000 (= 2^23 + 2^22), reinterpret as int, subtract, and you have a round-to-nearest-integer — but only if |z| <= 2^22.
z + (2^23 + 2^22) is representable in [2^22, 2^23], so the fraction bits
encode the integer part. Outside that range, the bit trick produces garbage.
For most activation functions, z is naturally bounded. But softplus_exp_negative passed z unclamped to the helper, and for |x| >= ~8.7e6, |z| = |x|/ln(2) > 2^22, so:
k_int instead of a large negative one.new_exp = p_exp + k_int becomes large and positive.new_exp > 0 flush-to-zero guard (meant for underflow) sees a positive exponent and writes it straight into the 8-bit exponent field.+inf or NaN.
// Before:
sfpi::vFloat z = x * INV_LN2;
sfpi::vFloat k = _sfpu_round_to_nearest_int32_(z, k_int); // 💥 z unbounded
// After:
sfpi::vFloat z = x * INV_LN2;
constexpr float UNDERFLOW_THRESHOLD = -126.5f;
z = sfpi::max(z, UNDERFLOW_THRESHOLD); // ✅ matches xielu, gelu, etc.
sfpi::vFloat k = _sfpi_round_to_nearest_int32_(z, k_int);
The clamp is exact because exp(x) underflows to 0 for x < -126.5 in float32. Clamping z to -126.5 means k_int ≈ -126, which gives new_exp < 0, so the flush-to-zero guard correctly returns 0 — exactly what softplus should return for large negative inputs.
Modern AI accelerators push floating-point to its limits:
2^-126 to 2^126
Every other eltwise op in the codebase already clamps — xielu, gelu, exp, sigmoid all bound their arguments to the rounding helper. softplus was the one that didn't.
The same class of bug appears in ttnn.reciprocal (issue #55797): the Blackhole fp32 path uses additive Newton-Raphson refinement (y = t2*y + y), which underflows for |x| >= 2^119. The multiplicative form ( y = y * (2 - x*y)) used by rdiv and pow doesn't have this problem.
The lesson: in subnormal-range arithmetic, the order of operations matters. Computing 1 + small first, then multiplying, preserves precision that small * large + large loses.
When you're debugging a chip that costs more than most cars:
And yes — I'm hiring my debugging process as a service. Contact me on GitHub @truongsontung.