Show HN: Triton-blackhole – debug Triton FP drift without device_print

Triton-blackhole, a deterministic numerical debugger for Triton kernels, identifies where output diverges from a PyTorch reference and classifies drift as benign fp16/bf16 drift or a real bug without using tl.device_print or TRITON_INTERPRET. The tool, available via pip install, provides functions like compare, bisect_axes, bisect_tiles, classify_drift, and ProbeBank to pinpoint hotspots, minimal failing regions, and fusion-boundary stage diffs. It supports Linux, WSL2 with NVIDIA, and Colab GPU, but not native Windows for live Triton kernels.

torch.allclose failed. Now what? triton-blackhole is a deterministic numerical debugger for Triton kernels. It finds where your output diverges from a PyTorch reference — and whether it's benign fp16/bf16 drift or a real bug — without tl.device print floods or TRITON INTERPRET . python from triton blackhole import diagnose print diagnose triton out, torch ref → hotspot index, drift kind, minimal failing region ▶ Open the 2‑minute Colab demo Runtime → GPU | You try… | What happens | |---|---| torch.allclose ... | False . No index. No cause. | tl.device print | Thousands of unsynced lines. No tensor context. | TRITON INTERPRET=1 | Breaks on bf16 / tl.load tl.load ... . Not your real kernel. | | Tool | What you get | |---|---| compare | Max-error indices, hotspots, neighborhoods, bf16-aware tolerances | bisect axes | Minimal failing sub-tensor | bisect tiles | Failing program id range on the compiled kernel | classify drift | localized bug vs reduction order vs dtype cast vs … | ProbeBank | Fusion-boundary stage diffs not print spam | python from triton blackhole import compare, bisect axes, classify drift, format report from triton blackhole.classify import format classification print format report compare triton out, torch ref print format classification classify drift triton out, torch ref print bisect axes triton out, torch ref .report pip install triton-blackhole Optional Linux / WSL2 / Colab with NVIDIA : pip install triton-blackhole triton On Colab, pin Triton to whatever torch wants, e.g.: pip install "triton==3.6.0" From source: pip install -e ". dev " | Piece | Native Windows | WSL2 + NVIDIA | Linux + NVIDIA | Colab GPU | |---|---|---|---|---| | Debugger compare / bisect / classify | ✅ | ✅ | ✅ | ✅ | | Live Triton kernels | ❌ | ✅ | ✅ | ✅ | No NVIDIA laptop? Use the Colab badge above. python from triton blackhole import bisect tiles def launch pid lo, pid hi : return run triton kernel ..., pid lo=pid lo, pid hi=pid hi print bisect tiles launch, torch ref, num programs=grid .report python from triton blackhole.probe import ProbeBank bank = ProbeBank bank.capture "pre softmax", scores ref, side="ref" bank.capture "pre softmax", scores tri, side="tri" print bank.report first diverging stage triton-blackhole compare triton out.pt torch ref.pt --bisect --suggest We never interpret the kernel. We bisect: Output space tensor axes Grid space program id Stage space named intermediates Same binary as production — bf16, tensor cores, indirect loads included. python examples/demo softmax drift.py python examples/demo tile bisect.py python examples/demo triton add.py needs CUDA + triton