{"slug": "show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes", "title": "Show HN: Fast NF4 dequantization Triton kernel (1.41x faster than bitsandbytes)", "summary": "A developer released an open-source Triton GPU kernel that dequantizes NF4 4-bit weights 1.27x–1.72x faster than the existing bitsandbytes C++ library, using a single fused kernel with inline PTX assembly and L2 cache eviction optimizations. The kernel eliminates CPU dispatch overhead and supports arbitrary tensor sizes, BF16/FP16 output, and torch.compile compatibility.", "body_md": "**Optimized NF4 (NormalFloat 4-bit) dequantization via Triton GPU kernels**\n\nUnsloth AI Founding Engineer Challenge #1— Convert NF4/BnB 4-bit dequantization from C++ to Triton. Achieves1.27x–1.41x speedupover`bitsandbytes`\n\nC++ across all tensor sizes.\n\n```\npip install git+https://github.com/Griffith-7/nf4-triton-kernel.git\npython\nfrom nf4_kernel import dequant_nf4\n\noutput = dequant_nf4(packed_weights, absmax_scales, dtype=torch.bfloat16)\n```\n\nLLMs use 4-bit NF4 quantization to fit in VRAM. At inference time, weights must be **dequantized** back to 16-bit. The existing `bitsandbytes`\n\nlibrary does this in C++, but suffers from CPU dispatch overhead. This project implements the same operation as a **single fused Triton kernel** — eliminating the overhead and achieving consistent speedups.\n\nTested against `bitsandbytes`\n\nv0.49.2 on **RTX 3050 Laptop GPU** (PyTorch 2.13.0, Triton 3.6.0):\n\n| Tensor Size | Triton Kernel | bitsandbytes C++ | Speedup |\n|---|---|---|---|\n| 4,096 | 0.021 ms | 0.030 ms | 1.43x |\n| 16,384 | 0.018 ms | 0.030 ms | 1.68x |\n| 65,536 | 0.017 ms | 0.029 ms | 1.72x |\n| 262,144 | 0.025 ms | 0.035 ms | 1.39x |\n\nAll sizes pass the **>1.15x** threshold. On Tesla T4, the original author reported **2.09x**.\n\nA single fused Triton kernel performs three operations in one GPU pass:\n\n```\nuint8 packed bytes → bit unpack → NF4 table lookup (PTX ASM) → absmax scale → BF16/FP16 output\n```\n\n**Bit unpacking**— splits each`uint8`\n\ninto two 4-bit nibbles via bit-shifting**NF4 lookup**— maps each 4-bit index to a float using inline PTX assembly (16 values hardcoded in GPU registers)** Absmax scaling**— multiplies by block-wise scaling factor and stores as BF16 or FP16\n\n| Optimization | Detail |\n|---|---|\nInline PTX Assembly |\nNF4 lookup table lives in GPU registers via `tl.inline_asm_elementwise` — zero memory reads, no branch mispredictions |\nL2 Cache Eviction |\n`evict_first` for streaming packed weights, `evict_last` for shared absmax — prevents cache thrashing |\nSingle Fused Kernel |\nBit-unpack + lookup + scale in one pass — no intermediate memory round-trips |\nLarge Block Size |\n1024 elements per thread block for optimal GPU occupancy |\nDynamic Shapes |\nHandles arbitrary tensor sizes without hardcoded grid/block dimensions |\n\n| Feature | Status | Points |\n|---|---|---|\n| Single Fused Kernel | Passed | +3 |\n| Inline PTX Assembly | Passed | +3 |\n| L2 Cache Eviction | Passed | +1 |\n| BF16/FP16 Output | Passed | +1 |\n| torch.compile Compatible | Passed (inductor) | +1 |\n| >1.15x Speedup vs BnB | Passed (1.39x – 1.72x) | +5 |\nTotal |\n14/14 |\n\n```\n# From GitHub\npip install git+https://github.com/Griffith-7/nf4-triton-kernel.git\n\n# Editable (dev)\ngit clone https://github.com/Griffith-7/nf4-triton-kernel.git\ncd nf4-triton-kernel\npip install -e \".[dev-all]\"\n```\n\n- Python 3.10+\n- PyTorch 2.13+\n- Triton 3.6+\n- CUDA 12.4+ GPU\n\n```\nnf4-triton-kernel/\n├── src/nf4_kernel/\n│   ├── __init__.py          # Package exports, version\n│   └── kernel.py            # Core Triton kernel + quantize/dequantize utils\n├── tests/\n│   └── test_nf4_kernel.py   # 21 unit tests (correctness, edge cases, validation)\n├── benchmarks/\n│   └── benchmark.py         # Performance benchmark vs bitsandbytes\n├── .github/workflows/\n│   └── ci.yml               # GitHub Actions CI (lint + test)\n├── pyproject.toml           # PEP 621 project config\n├── .pre-commit-config.yaml  # Pre-commit hooks (ruff, codespell, etc.)\n├── Dockerfile               # Multi-stage build (CPU + CUDA)\n├── colab_notebook.py        # Google Colab notebook (cell-by-cell)\n└── README.md\npython\nimport torch\nfrom nf4_kernel import dequant_nf4\n\n# packed_weights: uint8 tensor (two 4-bit values per byte)\n# absmax: block-wise scaling factors (FP8 or float32)\noutput = dequant_nf4(packed_weights, absmax, group_size=64, dtype=torch.bfloat16)\noutput_fp16 = dequant_nf4(packed_weights, absmax, dtype=torch.float16)\npython\n@torch.compile\ndef compiled_dequant(pw, am):\n    return dequant_nf4(pw, am)\n\noutput = compiled_dequant(packed_weights, absmax)\npython -m pytest tests/ -v\n```\n\n21 tests covering:\n\n- Functional correctness (BF16, FP16)\n- Edge cases (small/large/odd/non-aligned tensors, zeros, uniform, alternating)\n- Quantize utility validation\n- Input validation errors\n- Multiple group sizes (32, 64, 128)\n\n```\n!git clone https://github.com/Griffith-7/nf4-triton-kernel.git\n%cd nf4-triton-kernel\npip install -e .\nexec(open(\"colab_notebook.py\").read())\n```\n\nMIT License — see [LICENSE](/Griffith-7/nf4-triton-kernel/blob/main/LICENSE).", "url": "https://wpnews.pro/news/show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes", "canonical_source": "https://github.com/Griffith-7/nf4-triton-kernel", "published_at": "2026-07-15 13:41:55+00:00", "updated_at": "2026-07-15 13:48:02.079715+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["Unsloth AI", "bitsandbytes", "Triton", "PyTorch", "NVIDIA", "RTX 3050", "Tesla T4", "Griffith-7"], "alternates": {"html": "https://wpnews.pro/news/show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes", "markdown": "https://wpnews.pro/news/show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes.md", "text": "https://wpnews.pro/news/show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes.txt", "jsonld": "https://wpnews.pro/news/show-hn-fast-nf4-dequantization-triton-kernel-1-41x-faster-than-bitsandbytes.jsonld"}}