{"slug": "show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print", "title": "Show HN: Triton-blackhole – debug Triton FP drift without device_print", "summary": "Triton-blackhole, a deterministic numerical debugger for Triton kernels, identifies where output diverges from a PyTorch reference and classifies drift as benign fp16/bf16 drift or a real bug without using tl.device_print or TRITON_INTERPRET. The tool, available via pip install, provides functions like compare, bisect_axes, bisect_tiles, classify_drift, and ProbeBank to pinpoint hotspots, minimal failing regions, and fusion-boundary stage diffs. It supports Linux, WSL2 with NVIDIA, and Colab GPU, but not native Windows for live Triton kernels.", "body_md": "`torch.allclose`\n\nfailed. Now what?\n\ntriton-blackhole is a deterministic numerical debugger for Triton kernels. It finds *where* your output diverges from a PyTorch reference — and whether it's benign fp16/bf16 drift or a real bug — **without** `tl.device_print`\n\nfloods or `TRITON_INTERPRET`\n\n.\n\n``` python\nfrom triton_blackhole import diagnose\n\nprint(diagnose(triton_out, torch_ref))\n# → hotspot index, drift kind, minimal failing region\n```\n\n** ▶ Open the 2‑minute Colab demo** (Runtime → GPU)\n\n| You try… | What happens |\n|---|---|\n`torch.allclose(...)` |\n`False` . No index. No cause. |\n`tl.device_print` |\nThousands of unsynced lines. No tensor context. |\n`TRITON_INTERPRET=1` |\nBreaks on bf16 / `tl.load(tl.load(...))` . Not your real kernel. |\n\n| Tool | What you get |\n|---|---|\n`compare` |\nMax-error indices, hotspots, neighborhoods, bf16-aware tolerances |\n`bisect_axes` |\nMinimal failing sub-tensor |\n`bisect_tiles` |\nFailing `program_id` range on the compiled kernel |\n`classify_drift` |\n`localized_bug` vs `reduction_order` vs `dtype_cast` vs … |\n`ProbeBank` |\nFusion-boundary stage diffs (not print spam) |\n\n``` python\nfrom triton_blackhole import compare, bisect_axes, classify_drift, format_report\nfrom triton_blackhole.classify import format_classification\n\nprint(format_report(compare(triton_out, torch_ref)))\nprint(format_classification(classify_drift(triton_out, torch_ref)))\nprint(bisect_axes(triton_out, torch_ref).report())\npip install triton-blackhole\n```\n\nOptional (Linux / WSL2 / Colab with NVIDIA):\n\n```\npip install triton-blackhole[triton]\n# On Colab, pin Triton to whatever torch wants, e.g.:\n# pip install \"triton==3.6.0\"\n```\n\nFrom source:\n\n```\npip install -e \".[dev]\"\n```\n\n| Piece | Native Windows | WSL2 + NVIDIA | Linux + NVIDIA | Colab GPU |\n|---|---|---|---|---|\n| Debugger (compare / bisect / classify) | ✅ | ✅ | ✅ | ✅ |\n| Live Triton kernels | ❌ | ✅ | ✅ | ✅ |\n\nNo NVIDIA laptop? Use the Colab badge above.\n\n``` python\nfrom triton_blackhole import bisect_tiles\n\ndef launch(pid_lo, pid_hi):\n    return run_triton_kernel(..., pid_lo=pid_lo, pid_hi=pid_hi)\n\nprint(bisect_tiles(launch, torch_ref, num_programs=grid).report())\npython\nfrom triton_blackhole.probe import ProbeBank\n\nbank = ProbeBank()\nbank.capture(\"pre_softmax\", scores_ref, side=\"ref\")\nbank.capture(\"pre_softmax\", scores_tri, side=\"tri\")\nprint(bank.report())  # first diverging stage\ntriton-blackhole compare triton_out.pt torch_ref.pt --bisect --suggest\n```\n\nWe never interpret the kernel. We bisect:\n\n**Output space**(tensor axes)** Grid space**(`program_id`\n\n)**Stage space**(named intermediates)\n\nSame binary as production — bf16, tensor cores, indirect loads included.\n\n```\npython examples/demo_softmax_drift.py\npython examples/demo_tile_bisect.py\npython examples/demo_triton_add.py   # needs CUDA + triton\n```\n\n", "url": "https://wpnews.pro/news/show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print", "canonical_source": "https://github.com/brian-mwirigi/triton-blackhole", "published_at": "2026-07-23 16:05:36+00:00", "updated_at": "2026-07-23 16:22:53.913441+00:00", "lang": "en", "topics": ["developer-tools", "machine-learning", "artificial-intelligence"], "entities": ["triton-blackhole", "PyTorch", "Triton", "NVIDIA", "Colab"], "alternates": {"html": "https://wpnews.pro/news/show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print", "markdown": "https://wpnews.pro/news/show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print.md", "text": "https://wpnews.pro/news/show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print.txt", "jsonld": "https://wpnews.pro/news/show-hn-triton-blackhole-debug-triton-fp-drift-without-device-print.jsonld"}}