{"slug": "bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a", "title": "Bw24 – from scratch rust+CUDA inference, every kernel tuned for sm_120a", "summary": "A developer known as Avifenesh released Bw24, a from-scratch LLM inference engine in Rust and CUDA tuned for NVIDIA's RTX 5090 Laptop GPU (Blackwell sm_120a, 24 GB), achieving up to 2.3x speedup over llama.cpp via MTP speculative decoding while guaranteeing token-identical outputs. The engine supports six models including Qwen and Gemma variants, ships with prebuilt Linux binaries, and is designed for single-user, single-GPU serving on RTX 50-series cards.", "body_md": "From-scratch LLM inference engine in Rust + CUDA, built for one machine: an RTX 5090 Laptop (Blackwell sm_120a, 24 GB). No frameworks, no ggml — every kernel written and tuned against measured hardware limits, with llama.cpp as the benchmark to beat on the same rig.\n\nThe headline capability is **MTP speculative decoding**: up to 2.3x over llama.cpp's best spec config, leading on every supported Qwen model and prompt class (1.06-2.30x per cell), with trimmed drafter heads published ready-to-use ([huggingface.co/Avifenesh/bw24-bench](https://huggingface.co/Avifenesh/bw24-bench)) — behind a drop-in OpenAI-compatible server. Exactness is the contract: speculative output is gated token-identical to plain decode, so the speedup never changes what the model says.\n\n**Use bw24 when** you serve one model to one user on an RTX 50-series card and want measured, exactness-gated speed. **Use something else when** you have any other GPU ([llama.cpp](https://github.com/ggml-org/llama.cpp), [mistral.rs](https://github.com/EricLBuehler/mistral.rs)) or need multi-GPU / batched serving (vLLM, SGLang).\n\nRunning bw24 on your own rig — desktop 50-series, older NVIDIA, anything? A [hardware validation report](/avifenesh/bw24/blob/main/.github/ISSUE_TEMPLATE/hardware-validation.md) is the fastest way to help: 50-series reports bless the rest of the family, older-card reports map the compatibility floor.\n\n**Current standing: six supported models, all fully gated. Qwen leads llama.cpp on every cell (plain 1.06-1.08x, spec 1.06-2.30x). Gemma leads decisively where llama lacks the capability or the depth (31B spec 1.7k 1.16x, E4B spec ≥1.23x, E4B plain 1.10x) and sits at 0.99-1.06x elsewhere under the strictest best-vs-best pairing (2026-07-15 re-audit).** Every number below is a same-session, same-prompt, interleaved measurement against llama.cpp's best config; exactness is gated (argmax match + speculative self-consistency) on every kernel change, so speed never buys different outputs.\n\n| Tier | Models | State |\n|---|---|---|\nSupported |\nQwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE (NVFP4/IQ4_XS); Gemma-4 26B-A4B MoE, 31B dense, E4B (QAT Q4_0 + MTP drafters) | Board-published, fully gated, exactness-first; margins per model in the tables below |\nSupported, under tuning |\nHy3 Layer103.5 overlay (VRAM→RAM→dual-NVMe spill) | Runs end-to-end through bw24-native CPU/GPU serving, correctness-gated on the 5090 target; see\n|\n\n**In progress** Prebuilt Linux x86_64 binaries (sm_120a) ship with each [release](https://github.com/avifenesh/bw24/releases) — or build from source:\n\n```\ncargo build --release\n./target/release/kernel-check                     # every kernel vs CPU reference\nBW24_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt \"Explain KV caches.\"\nBW24_SPEC_K=3 ./target/release/run-spec /path/to/qwen36-27b.gguf   # MTP speculative\n./target/release/run-gen hf:owner/repo:Q4_K_M --prompt \"hi\"        # auto-download from HF (needs `hf` CLI)\n./target/release/frspec-owngen model.gguf trim.gguf --validate     # build + validate an FR-Spec draft trim from the model's OWN generations\n./target/release/bw24-server                      # OpenAI-compatible /v1\n```\n\nExpected output — `kernel-check`\n\nends with:\n\n```\nALL GREEN: kernels match CPU reference.\n```\n\nand `run-gen`\n\nprints its correctness gate before any generation:\n\n```\nverify-prefill argmax=N  decode argmax=N  logit maxdiff=...  MATCH\n```\n\nTuned paths are the defaults — no flags needed. Flags exist only for runtime parameters, machine config, and rollback seams (`docs/FLAGS.md`\n\n). A MISMATCH line from the gate voids every number after it.\n\nServing Hy3 (a ~100 GB expert bank) on this 24 GB card uses a frozen HBM resident set, a bounded host cache, and positioned dual-NVMe reads — runbook, ABI safety notes, and current gate results in [docs/HY3-SPILL.md](/avifenesh/bw24/blob/main/docs/HY3-SPILL.md).\nThe paired native AVX-VNNI Q2_K path raises the local N=32 median from 4.37 to 4.60 tok/s across three interleaved, correctness-identical pairs (+5.3% by arm medians; [receipt](/avifenesh/bw24/blob/main/research/per-expert-quant/evidence/local-5090-native-next-20260721/q2k-avxvnni-pair-win.md)).\n\nMeasured 2026-07-18 on the target rig (RTX 5090 Laptop, N=2+ medians, both engines interleaved in the same thermal window on the same rig, same exact prompts, no flags (tuned paths are defaults); plain/depth rows from the 2026-07-09 validity-gated cold-start rebaseline, spec rows re-paired 2026-07-18, Gemma card rows from the 2026-07-15 best-vs-best re-audit. Full per-run logs: research/tune-data/ (Qwen) and research/gemma4-bringup/ (Gemma) — every win and every loss) against llama.cpp built on the same machine, same exact prompts, both engines re-baselined the same day. Boards move with the tuning campaign — `research/tune-data/rig5090.jsonl`\n\nis the running record; the README is refreshed with every board-moving merge.\n\n**Plain decode** (no speculation, tg128 at 512-token context):\n\n| Model | bw24 plain | llama.cpp plain | Ratio |\n|---|---|---|---|\n| Qwen3.5-9B NVFP4 (GGUF) | 135.7 | 126.7 | 1.07x |\n| Qwen3.6-27B NVFP4 (GGUF) | 48.4 | 44.9 | 1.08x |\n| Qwen3.6-35B-A3B MoE (IQ4_XS) | 178.2 | 167.8 | 1.06x |\n\nDepth is part of the contract: at 6.3k-token context every lead holds (1.02-1.07x).\n\n**Speculative decoding** (MTP head, both engines at their measured best):\n\n| Model | bw24 spec | llama.cpp spec-best | Ratio |\n|---|---|---|---|\n| Qwen3.5-9B (K=3 + own-gen trimmed draft) | 281.0 / 211.7 / 187.1 | 122.2 / 121.5 / 117.7 | 2.30x / 1.74x / 1.59x |\n| Qwen3.6-27B (K=3 + own-gen trimmed draft) | 116.4 / 101.2 / 86.0 | 91.7 / 93.3 / 81.5 | 1.27x / 1.08x / 1.06x |\n| Qwen3.6-35B-A3B (K=2 + own-gen trimmed draft) | 280.6 / 259.6 / 258.0 | 236.5 / 174.6 / 173.5 | 1.19x / 1.49x / 1.49x |\n\nThe three columns are three prompt classes: short code / medium code (both greedy) / long agentic, sampled at temp 0.7 with distribution-exact rejection sampling. One asterisk the log carries: the 35B short-code llama bar (236.5) rode an EOS-margin flip and is not a clean win basis — the other two 35B cells are. Every spec row uses **one trimmed draft file built by the standard regime** — the model's own-generation FR-Spec ranks, byte-verbatim MTP extraction, NVFP4 head + Q4_K_M block ([ docs/DRAFT-REGIME.md](/avifenesh/bw24/blob/main/docs/DRAFT-REGIME.md)).\n\n**Drafts: use ours or build your own.** Prebuilt per-model drafts (exact pipeline, exact published bytes) live at [huggingface.co/Avifenesh/bw24-bench](https://huggingface.co/Avifenesh/bw24-bench) under `drafts/<model>/`\n\n. For any other model, requant, or finetune, build one in two commands (a finetune's distribution moved, so its draft must too):\n\n```\n./target/release/frspec-owngen model.gguf ranks.gguf 32768        # ranks from the model's OWN generations\ntools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf  # extract + trim + quantize\n```\n\nExact prompts and configs also in the bench repo; llama.cpp flags in [docs/COMPETITOR-SETUP.md](/avifenesh/bw24/blob/main/docs/COMPETITOR-SETUP.md).\n\nSame protocol, own campaign log (`research/gemma4-bringup/rig5090-gemma4.jsonl`\n\n); all cells re-paired 2026-07-15 under best-vs-best (llama server MTP at its swept-best flags, exact-token-id prompts, serialized same-window arms). Contexts are real prompt depths.\n\n**26B-A4B MoE:**\n\n| Cell | bw24 | llama.cpp | Ratio |\n|---|---|---|---|\n| plain, short ctx | 199.7 | 187.6 | 1.06x |\n| plain, 1.7k ctx | 183.1 | 173.1 | 1.06x |\n| plain, 4.9k ctx | 162.6 | 142.0 | 1.14x (stale fa-off bar — re-pair pending) |\n| MTP spec, short ctx (K=6) | 267 | 271 | 0.99x |\n| MTP spec, 1.7k ctx (K=6 + FR trim) | 298 | 286 | 1.04x |\n\n**31B dense / E4B:**\n\n| Cell | bw24 | llama.cpp | Ratio |\n|---|---|---|---|\n| 31B plain, short | 40.8 | 40.2 | 1.02x |\n| 31B plain, 1.7k | 38.4 | 37.4 | 1.03x |\n| 31B MTP spec, short (K=3) | 98.0 | 92.3 | 1.06x |\n| 31B MTP spec, 1.7k (K=6 + FR trim) | 97.3 | 83.9 | 1.16x |\n| E4B MTP spec (K=6 assistant) | 248 | no llama MTP for E4B — vs its plain 181.0 | ≥1.23x |\n| E4B plain, short | 199.9 | 181.0 | 1.10x |\n\nWhat buys the margins: an FP8 (e4m3) KV cache (half the bytes of llama's f16 KV at near-zero dequant cost), occupancy-tuned attention tiles, wide-load Q4_0 expert dots, and FR-Spec drafter-head trims (150 MB → 18 MB at unchanged 0.91-0.94 acceptance on the 26B). One structural finding worth knowing: FP8 noise in the *windowed* KV layers guts the MTP drafter's acceptance, so spec serving automatically keeps those layers at q8_0/q5_1 while plain serving keeps FP8 — a config discovery, not a kernel. The same FP8-KV lever is available for Qwen behind `BW24_KV_FP8`\n\n(correctness-proven, ~45% smaller KV). Since 2026-07-19 an adaptive serve-time trim lifts the 31B spec cells a further ~2-5% solo (~105 short / ~104 at 1.7k); those runs are not yet llama-re-paired, so the table keeps the paired ratios.\n\nEarlier published Gemma spec margins (1.37-1.54x, and the 31B 167.5/112.1 pair) do not reproduce — even through era binaries — and are retired; the campaign log carries the full archaeology. llama's spec side is `--spec-type draft-mtp`\n\n, warm, at its measured best on the same box; E4B has no llama MTP arm at all (the 2026-06-30 freeze binary can't serve its drafter — fixed upstream later), so its spec row is floored against llama's plain. Correctness is structural, not statistical: decode, verify, and graph replay launch the same kernel symbols, so the verify gate reads a bit-exact 0.000e0 logit maxdiff at every depth.\n\n**Reproducing:** exact-token-id prompts and llama.cpp's swept-best flags in [docs/COMPETITOR-SETUP.md](/avifenesh/bw24/blob/main/docs/COMPETITOR-SETUP.md); every row's raw run — and every retired number's archaeology — in [ research/gemma4-bringup/rig5090-gemma4.jsonl](/avifenesh/bw24/blob/main/research/gemma4-bringup/rig5090-gemma4.jsonl).\n\n**Prefill** trails llama.cpp (0.59-0.78x), root-caused: llama benches NVFP4 prefill at W4A4 (FP4 activations), a numeric class bw24's exactness gates reject — bw24's in-tree W4A4 arm beats llama but forks argmax on long prompts (`docs/FLAGS.md`\n\n§5). Output quality outranks the prefill column.- Gemma plain margins are thin where both engines sit at the DRAM wall (31B 1.02-1.03x, 26B 1.06x; best kernel = 91% of measured wall, e2e 87-89%). Every mechanism class measured — ours plus llama/vLLM/SGLang current releases — is shipped or carries a falsification row in the campaign log. Open spec cells: 31B short 1.06x, 26B 0.99x/1.04x.\n- Hy3 native spill is correctness-gated at a 4.60 tok/s N=3 median after the paired native AVX-VNNI Q2_K win and is being tuned toward a sustained 10 tok/s (\n[docs/HY3-SPILL.md](/avifenesh/bw24/blob/main/docs/HY3-SPILL.md)). - Safetensors runs checkpoints llama.cpp cannot (NVIDIA NVFP4 ST, 121 GB spilled MoEs) but GGUF is the primary delivery format — ST showed seed-sensitive long-context repetition (\n`research/tune-data/27b-st-vs-gguf-final.md`\n\n). The published Hy3 Layer103.5 expert overlay is the scoped exception.\n\n**NVFP4 / Q4_0 decode**— split-plane repacked matvecs, warp-level dp4a, int8 W4A8 tensor-core prefill GEMM, per-shape auto-dispatch.** MTP speculative decoding**— embedded draft head, one batched K+1 verify, zero-sync async rounds, adaptive draft depth; K=1..8 self-consistency gate.**MoE on 24 GB**— expert-major CSR batching, decode-once dequant, frozen SLRU expert residency, bounded host LRU, and mirrored positioned reads across VRAM→RAM→NVMe.**Quantized-KV attention**— fused prefill/decode FlashAttention-class kernels (q8_0/q5_1 or FP8-e4m3 KV per layer class), split-K, device-length counters for graph replay.**CUDA-graph decode**— one graph replay per token, 4 bytes/token host traffic.** Hybrid + sigmoid-router architectures**— gated-delta-net mixes (Qwen3.6), MiniMax/DeepSeek-style routing.** Safetensors loader**— modelopt NVFP4 repacks byte-exact; FP8/BF16 re-encode at load; disk-tier expert streaming.\n\nEvery kernel change passes, in order: `kernel-check`\n\n(CPU reference), the `run-gen`\n\nargmax gate, `run-spec`\n\nK=1..8 self-consistency — one command: `tools/local-ci.sh`\n\n. FP summation order is part of the contract — \"faster\" kernels that reduce in a different order get rejected when they flip argmax at tight margins.\n\nExactness gates are structurally blind to numeric shifts where decode and verify move *together* — that class silently cost half a spec margin across ~40 green commits in July 2026. The local perf CI (`tools/local-ci.sh --perf`\n\n) closes it: every published cell re-measured per engine-touching push, speculative **acceptance and tokens/round tracked per cell** against a rolling baseline (`research/tune-data/perf-ci.jsonl`\n\n), enforced by the pre-push hook. Upstream engines are swept weekly for portable decode mechanisms (`tools/upstream-sweep.sh`\n\n→ `research/upstream-sweeps.md`\n\n).\n\n| Crate | What it does |\n|---|---|\n`bw24-engine` |\nCUDA kernels (`cu/` ), forward passes, speculative decoding, MoE cache, graph decode |\n`bw24-gguf` |\nGGUF parser + tensor loading (memory-mapped) |\n`bw24-tokenizer` |\nBPE tokenizer + chat templates from GGUF metadata |\n`bw24-runtime` |\nCUDA device/stream/memory primitives over cudarc |\n`bw24-server` |\nOpenAI-compatible HTTP server (axum) |\n`bw24-probe` |\nStandalone hardware microbenches |\n\n- NVIDIA Blackwell consumer GPU (sm_120a); primary target RTX 5090 Laptop.\n- CUDA 13.1, plus 12.8 for the dual-toolkit build documented in\n[ARCHITECTURE.md](/avifenesh/bw24/blob/main/ARCHITECTURE.md)(`BW24_NVCC`\n\noverrides the nvcc path). Rust edition 2024, cudarc 0.19. - A model: GGUF or HF safetensors directory (pass either path). The optional Hy3 CPU-expert companion additionally needs a C++17 compiler with OpenMP.\n\n- Built for sm_120a only; tuning assumes this exact memory/compute ratio. On any other GPU,\nuse\n[llama.cpp](https://github.com/ggml-org/llama.cpp)(broadest hardware coverage) or[mistral.rs](https://github.com/EricLBuehler/mistral.rs)(multi-platform Rust) instead — a`feat/portable-ada-correctness`\n\nbranch exists for Ada (sm_89) but is untuned and its L40S lane is closed. - Single GPU, single stream; no tensor parallelism or continuous batching.\n- Moving research codebase; APIs and flags change without notice.\n\n— tech stack + sm_120a feasibility ledger.`ARCHITECTURE.md`\n\n— living state-of-work (standings, laws, open lanes).`HANDOVER.md`\n\n— design decision records.`docs/decisions/`\n\n— competitor engines at their peak on this box.`docs/COMPETITOR-SETUP.md`\n\n— Hy3 spill runbook +`docs/HY3-SPILL.md`\n\n[overlay release](/avifenesh/bw24/blob/main/research/per-expert-quant/hy3-layer103p5-release.md).+`research/tune-data/`\n\n— every experiment as JSONL, wins and losses both.`research/gemma4-bringup/`\n\n— the A/B measurement protocol.`research/benchmarks.md`\n\nIssues and PRs welcome — see [CONTRIBUTING.md](/avifenesh/bw24/blob/main/CONTRIBUTING.md).\n\nMIT — see [LICENSE](/avifenesh/bw24/blob/main/LICENSE).", "url": "https://wpnews.pro/news/bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a", "canonical_source": "https://github.com/avifenesh/bw24", "published_at": "2026-07-22 13:01:26+00:00", "updated_at": "2026-07-22 13:23:00.126734+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Avifenesh", "Bw24", "NVIDIA RTX 5090 Laptop", "llama.cpp", "Qwen", "Gemma", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a", "markdown": "https://wpnews.pro/news/bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a.md", "text": "https://wpnews.pro/news/bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a.txt", "jsonld": "https://wpnews.pro/news/bw24-from-scratch-rust-cuda-inference-every-kernel-tuned-for-sm-120a.jsonld"}}