# Bw24 – from scratch rust+CUDA inference, every kernel tuned for sm_120a

> Source: <https://github.com/avifenesh/bw24>
> Published: 2026-07-22 13:01:26+00:00

From-scratch LLM inference engine in Rust + CUDA, built for one machine: an RTX 5090 Laptop (Blackwell sm_120a, 24 GB). No frameworks, no ggml — every kernel written and tuned against measured hardware limits, with llama.cpp as the benchmark to beat on the same rig.

The headline capability is **MTP speculative decoding**: up to 2.3x over llama.cpp's best spec config, leading on every supported Qwen model and prompt class (1.06-2.30x per cell), with trimmed drafter heads published ready-to-use ([huggingface.co/Avifenesh/bw24-bench](https://huggingface.co/Avifenesh/bw24-bench)) — behind a drop-in OpenAI-compatible server. Exactness is the contract: speculative output is gated token-identical to plain decode, so the speedup never changes what the model says.

**Use bw24 when** you serve one model to one user on an RTX 50-series card and want measured, exactness-gated speed. **Use something else when** you have any other GPU ([llama.cpp](https://github.com/ggml-org/llama.cpp), [mistral.rs](https://github.com/EricLBuehler/mistral.rs)) or need multi-GPU / batched serving (vLLM, SGLang).

Running bw24 on your own rig — desktop 50-series, older NVIDIA, anything? A [hardware validation report](/avifenesh/bw24/blob/main/.github/ISSUE_TEMPLATE/hardware-validation.md) is the fastest way to help: 50-series reports bless the rest of the family, older-card reports map the compatibility floor.

**Current standing: six supported models, all fully gated. Qwen leads llama.cpp on every cell (plain 1.06-1.08x, spec 1.06-2.30x). Gemma leads decisively where llama lacks the capability or the depth (31B spec 1.7k 1.16x, E4B spec ≥1.23x, E4B plain 1.10x) and sits at 0.99-1.06x elsewhere under the strictest best-vs-best pairing (2026-07-15 re-audit).** Every number below is a same-session, same-prompt, interleaved measurement against llama.cpp's best config; exactness is gated (argmax match + speculative self-consistency) on every kernel change, so speed never buys different outputs.

| Tier | Models | State |
|---|---|---|
Supported |
Qwen3.5-9B, Qwen3.6-27B, Qwen3.6-35B-A3B MoE (NVFP4/IQ4_XS); Gemma-4 26B-A4B MoE, 31B dense, E4B (QAT Q4_0 + MTP drafters) | Board-published, fully gated, exactness-first; margins per model in the tables below |
Supported, under tuning |
Hy3 Layer103.5 overlay (VRAM→RAM→dual-NVMe spill) | Runs end-to-end through bw24-native CPU/GPU serving, correctness-gated on the 5090 target; see
|

**In progress** Prebuilt Linux x86_64 binaries (sm_120a) ship with each [release](https://github.com/avifenesh/bw24/releases) — or build from source:

```
cargo build --release
./target/release/kernel-check                     # every kernel vs CPU reference
BW24_CHAT=1 ./target/release/run-gen /path/to/model.gguf --prompt "Explain KV caches."
BW24_SPEC_K=3 ./target/release/run-spec /path/to/qwen36-27b.gguf   # MTP speculative
./target/release/run-gen hf:owner/repo:Q4_K_M --prompt "hi"        # auto-download from HF (needs `hf` CLI)
./target/release/frspec-owngen model.gguf trim.gguf --validate     # build + validate an FR-Spec draft trim from the model's OWN generations
./target/release/bw24-server                      # OpenAI-compatible /v1
```

Expected output — `kernel-check`

ends with:

```
ALL GREEN: kernels match CPU reference.
```

and `run-gen`

prints its correctness gate before any generation:

```
verify-prefill argmax=N  decode argmax=N  logit maxdiff=...  MATCH
```

Tuned paths are the defaults — no flags needed. Flags exist only for runtime parameters, machine config, and rollback seams (`docs/FLAGS.md`

). A MISMATCH line from the gate voids every number after it.

Serving Hy3 (a ~100 GB expert bank) on this 24 GB card uses a frozen HBM resident set, a bounded host cache, and positioned dual-NVMe reads — runbook, ABI safety notes, and current gate results in [docs/HY3-SPILL.md](/avifenesh/bw24/blob/main/docs/HY3-SPILL.md).
The paired native AVX-VNNI Q2_K path raises the local N=32 median from 4.37 to 4.60 tok/s across three interleaved, correctness-identical pairs (+5.3% by arm medians; [receipt](/avifenesh/bw24/blob/main/research/per-expert-quant/evidence/local-5090-native-next-20260721/q2k-avxvnni-pair-win.md)).

Measured 2026-07-18 on the target rig (RTX 5090 Laptop, N=2+ medians, both engines interleaved in the same thermal window on the same rig, same exact prompts, no flags (tuned paths are defaults); plain/depth rows from the 2026-07-09 validity-gated cold-start rebaseline, spec rows re-paired 2026-07-18, Gemma card rows from the 2026-07-15 best-vs-best re-audit. Full per-run logs: research/tune-data/ (Qwen) and research/gemma4-bringup/ (Gemma) — every win and every loss) against llama.cpp built on the same machine, same exact prompts, both engines re-baselined the same day. Boards move with the tuning campaign — `research/tune-data/rig5090.jsonl`

is the running record; the README is refreshed with every board-moving merge.

**Plain decode** (no speculation, tg128 at 512-token context):

| Model | bw24 plain | llama.cpp plain | Ratio |
|---|---|---|---|
| Qwen3.5-9B NVFP4 (GGUF) | 135.7 | 126.7 | 1.07x |
| Qwen3.6-27B NVFP4 (GGUF) | 48.4 | 44.9 | 1.08x |
| Qwen3.6-35B-A3B MoE (IQ4_XS) | 178.2 | 167.8 | 1.06x |

Depth is part of the contract: at 6.3k-token context every lead holds (1.02-1.07x).

**Speculative decoding** (MTP head, both engines at their measured best):

| Model | bw24 spec | llama.cpp spec-best | Ratio |
|---|---|---|---|
| Qwen3.5-9B (K=3 + own-gen trimmed draft) | 281.0 / 211.7 / 187.1 | 122.2 / 121.5 / 117.7 | 2.30x / 1.74x / 1.59x |
| Qwen3.6-27B (K=3 + own-gen trimmed draft) | 116.4 / 101.2 / 86.0 | 91.7 / 93.3 / 81.5 | 1.27x / 1.08x / 1.06x |
| Qwen3.6-35B-A3B (K=2 + own-gen trimmed draft) | 280.6 / 259.6 / 258.0 | 236.5 / 174.6 / 173.5 | 1.19x / 1.49x / 1.49x |

The three columns are three prompt classes: short code / medium code (both greedy) / long agentic, sampled at temp 0.7 with distribution-exact rejection sampling. One asterisk the log carries: the 35B short-code llama bar (236.5) rode an EOS-margin flip and is not a clean win basis — the other two 35B cells are. Every spec row uses **one trimmed draft file built by the standard regime** — the model's own-generation FR-Spec ranks, byte-verbatim MTP extraction, NVFP4 head + Q4_K_M block ([ docs/DRAFT-REGIME.md](/avifenesh/bw24/blob/main/docs/DRAFT-REGIME.md)).

**Drafts: use ours or build your own.** Prebuilt per-model drafts (exact pipeline, exact published bytes) live at [huggingface.co/Avifenesh/bw24-bench](https://huggingface.co/Avifenesh/bw24-bench) under `drafts/<model>/`

. For any other model, requant, or finetune, build one in two commands (a finetune's distribution moved, so its draft must too):

```
./target/release/frspec-owngen model.gguf ranks.gguf 32768        # ranks from the model's OWN generations
tools/make-trimmed-draft.sh model.gguf ranks.gguf.txt draft.gguf  # extract + trim + quantize
```

Exact prompts and configs also in the bench repo; llama.cpp flags in [docs/COMPETITOR-SETUP.md](/avifenesh/bw24/blob/main/docs/COMPETITOR-SETUP.md).

Same protocol, own campaign log (`research/gemma4-bringup/rig5090-gemma4.jsonl`

); all cells re-paired 2026-07-15 under best-vs-best (llama server MTP at its swept-best flags, exact-token-id prompts, serialized same-window arms). Contexts are real prompt depths.

**26B-A4B MoE:**

| Cell | bw24 | llama.cpp | Ratio |
|---|---|---|---|
| plain, short ctx | 199.7 | 187.6 | 1.06x |
| plain, 1.7k ctx | 183.1 | 173.1 | 1.06x |
| plain, 4.9k ctx | 162.6 | 142.0 | 1.14x (stale fa-off bar — re-pair pending) |
| MTP spec, short ctx (K=6) | 267 | 271 | 0.99x |
| MTP spec, 1.7k ctx (K=6 + FR trim) | 298 | 286 | 1.04x |

**31B dense / E4B:**

| Cell | bw24 | llama.cpp | Ratio |
|---|---|---|---|
| 31B plain, short | 40.8 | 40.2 | 1.02x |
| 31B plain, 1.7k | 38.4 | 37.4 | 1.03x |
| 31B MTP spec, short (K=3) | 98.0 | 92.3 | 1.06x |
| 31B MTP spec, 1.7k (K=6 + FR trim) | 97.3 | 83.9 | 1.16x |
| E4B MTP spec (K=6 assistant) | 248 | no llama MTP for E4B — vs its plain 181.0 | ≥1.23x |
| E4B plain, short | 199.9 | 181.0 | 1.10x |

What buys the margins: an FP8 (e4m3) KV cache (half the bytes of llama's f16 KV at near-zero dequant cost), occupancy-tuned attention tiles, wide-load Q4_0 expert dots, and FR-Spec drafter-head trims (150 MB → 18 MB at unchanged 0.91-0.94 acceptance on the 26B). One structural finding worth knowing: FP8 noise in the *windowed* KV layers guts the MTP drafter's acceptance, so spec serving automatically keeps those layers at q8_0/q5_1 while plain serving keeps FP8 — a config discovery, not a kernel. The same FP8-KV lever is available for Qwen behind `BW24_KV_FP8`

(correctness-proven, ~45% smaller KV). Since 2026-07-19 an adaptive serve-time trim lifts the 31B spec cells a further ~2-5% solo (~105 short / ~104 at 1.7k); those runs are not yet llama-re-paired, so the table keeps the paired ratios.

Earlier published Gemma spec margins (1.37-1.54x, and the 31B 167.5/112.1 pair) do not reproduce — even through era binaries — and are retired; the campaign log carries the full archaeology. llama's spec side is `--spec-type draft-mtp`

, warm, at its measured best on the same box; E4B has no llama MTP arm at all (the 2026-06-30 freeze binary can't serve its drafter — fixed upstream later), so its spec row is floored against llama's plain. Correctness is structural, not statistical: decode, verify, and graph replay launch the same kernel symbols, so the verify gate reads a bit-exact 0.000e0 logit maxdiff at every depth.

**Reproducing:** exact-token-id prompts and llama.cpp's swept-best flags in [docs/COMPETITOR-SETUP.md](/avifenesh/bw24/blob/main/docs/COMPETITOR-SETUP.md); every row's raw run — and every retired number's archaeology — in [ research/gemma4-bringup/rig5090-gemma4.jsonl](/avifenesh/bw24/blob/main/research/gemma4-bringup/rig5090-gemma4.jsonl).

**Prefill** trails llama.cpp (0.59-0.78x), root-caused: llama benches NVFP4 prefill at W4A4 (FP4 activations), a numeric class bw24's exactness gates reject — bw24's in-tree W4A4 arm beats llama but forks argmax on long prompts (`docs/FLAGS.md`

§5). Output quality outranks the prefill column.- Gemma plain margins are thin where both engines sit at the DRAM wall (31B 1.02-1.03x, 26B 1.06x; best kernel = 91% of measured wall, e2e 87-89%). Every mechanism class measured — ours plus llama/vLLM/SGLang current releases — is shipped or carries a falsification row in the campaign log. Open spec cells: 31B short 1.06x, 26B 0.99x/1.04x.
- Hy3 native spill is correctness-gated at a 4.60 tok/s N=3 median after the paired native AVX-VNNI Q2_K win and is being tuned toward a sustained 10 tok/s (
[docs/HY3-SPILL.md](/avifenesh/bw24/blob/main/docs/HY3-SPILL.md)). - Safetensors runs checkpoints llama.cpp cannot (NVIDIA NVFP4 ST, 121 GB spilled MoEs) but GGUF is the primary delivery format — ST showed seed-sensitive long-context repetition (
`research/tune-data/27b-st-vs-gguf-final.md`

). The published Hy3 Layer103.5 expert overlay is the scoped exception.

**NVFP4 / Q4_0 decode**— split-plane repacked matvecs, warp-level dp4a, int8 W4A8 tensor-core prefill GEMM, per-shape auto-dispatch.** MTP speculative decoding**— embedded draft head, one batched K+1 verify, zero-sync async rounds, adaptive draft depth; K=1..8 self-consistency gate.**MoE on 24 GB**— expert-major CSR batching, decode-once dequant, frozen SLRU expert residency, bounded host LRU, and mirrored positioned reads across VRAM→RAM→NVMe.**Quantized-KV attention**— fused prefill/decode FlashAttention-class kernels (q8_0/q5_1 or FP8-e4m3 KV per layer class), split-K, device-length counters for graph replay.**CUDA-graph decode**— one graph replay per token, 4 bytes/token host traffic.** Hybrid + sigmoid-router architectures**— gated-delta-net mixes (Qwen3.6), MiniMax/DeepSeek-style routing.** Safetensors loader**— modelopt NVFP4 repacks byte-exact; FP8/BF16 re-encode at load; disk-tier expert streaming.

Every kernel change passes, in order: `kernel-check`

(CPU reference), the `run-gen`

argmax gate, `run-spec`

K=1..8 self-consistency — one command: `tools/local-ci.sh`

. FP summation order is part of the contract — "faster" kernels that reduce in a different order get rejected when they flip argmax at tight margins.

Exactness gates are structurally blind to numeric shifts where decode and verify move *together* — that class silently cost half a spec margin across ~40 green commits in July 2026. The local perf CI (`tools/local-ci.sh --perf`

) closes it: every published cell re-measured per engine-touching push, speculative **acceptance and tokens/round tracked per cell** against a rolling baseline (`research/tune-data/perf-ci.jsonl`

), enforced by the pre-push hook. Upstream engines are swept weekly for portable decode mechanisms (`tools/upstream-sweep.sh`

→ `research/upstream-sweeps.md`

).

| Crate | What it does |
|---|---|
`bw24-engine` |
CUDA kernels (`cu/` ), forward passes, speculative decoding, MoE cache, graph decode |
`bw24-gguf` |
GGUF parser + tensor loading (memory-mapped) |
`bw24-tokenizer` |
BPE tokenizer + chat templates from GGUF metadata |
`bw24-runtime` |
CUDA device/stream/memory primitives over cudarc |
`bw24-server` |
OpenAI-compatible HTTP server (axum) |
`bw24-probe` |
Standalone hardware microbenches |

- NVIDIA Blackwell consumer GPU (sm_120a); primary target RTX 5090 Laptop.
- CUDA 13.1, plus 12.8 for the dual-toolkit build documented in
[ARCHITECTURE.md](/avifenesh/bw24/blob/main/ARCHITECTURE.md)(`BW24_NVCC`

overrides the nvcc path). Rust edition 2024, cudarc 0.19. - A model: GGUF or HF safetensors directory (pass either path). The optional Hy3 CPU-expert companion additionally needs a C++17 compiler with OpenMP.

- Built for sm_120a only; tuning assumes this exact memory/compute ratio. On any other GPU,
use
[llama.cpp](https://github.com/ggml-org/llama.cpp)(broadest hardware coverage) or[mistral.rs](https://github.com/EricLBuehler/mistral.rs)(multi-platform Rust) instead — a`feat/portable-ada-correctness`

branch exists for Ada (sm_89) but is untuned and its L40S lane is closed. - Single GPU, single stream; no tensor parallelism or continuous batching.
- Moving research codebase; APIs and flags change without notice.

— tech stack + sm_120a feasibility ledger.`ARCHITECTURE.md`

— living state-of-work (standings, laws, open lanes).`HANDOVER.md`

— design decision records.`docs/decisions/`

— competitor engines at their peak on this box.`docs/COMPETITOR-SETUP.md`

— Hy3 spill runbook +`docs/HY3-SPILL.md`

[overlay release](/avifenesh/bw24/blob/main/research/per-expert-quant/hy3-layer103p5-release.md).+`research/tune-data/`

— every experiment as JSONL, wins and losses both.`research/gemma4-bringup/`

— the A/B measurement protocol.`research/benchmarks.md`

Issues and PRs welcome — see [CONTRIBUTING.md](/avifenesh/bw24/blob/main/CONTRIBUTING.md).

MIT — see [LICENSE](/avifenesh/bw24/blob/main/LICENSE).