Fusing a 27B ternary LLM's whole decode step into one CUDA kernel

Infra AI open-sourced bonsai-turbo, a batch-1 decode engine that runs PrismML's Bonsai 27B ternary LLM 1.76x faster than the official llama.cpp fork on an H100, achieving 151 tok/s (ternary) and 159 tok/s (1-bit) with exact logit parity. The speedup comes from fusing the entire per-token decode step into a single CUDA kernel, eliminating GPU op overhead that previously consumed 97% of execution time.

i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token H100, tg128, greedy: ternary 85.5 151 tok/s. 1-bit 90.1 159 tok/s. logit parity with the fork on 32 of 32 test prompts, gated before any speed number counts. not a lossy trick why it's faster: at batch-1 the GPU isn't math-bound or bandwidth-bound, it's overhead-bound. the stock path executes 3703 GPU ops per token and spends ~97% of its time on that op overhead. bonsai-turbo fuses the whole per-token pass into a handful of large ops. --mega mode compiles the entire 64-layer token step -- embed layers logits next token -- into one cooperative kernel the kernels were generated by our internal agent -- the same agent that powers @runinfrai scope is deliberate: batch-1 decode only. Bonsai 27B is the ternary 27B small enough to run on a phone, and this makes the local single-user experience actually fast. it is not a batched-serving engine roofline says ~440-490 tok/s is on the table. next: cp.async weight pipelining, then a speculative drafter -- targeting ~300 github.com/RightNow-AI/bo…