{"slug": "fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel", "title": "Fusing a 27B ternary LLM's whole decode step into one CUDA kernel", "summary": "Infra AI open-sourced bonsai-turbo, a batch-1 decode engine that runs PrismML's Bonsai 27B ternary LLM 1.76x faster than the official llama.cpp fork on an H100, achieving 151 tok/s (ternary) and 159 tok/s (1-bit) with exact logit parity. The speedup comes from fusing the entire per-token decode step into a single CUDA kernel, eliminating GPU op overhead that previously consumed 97% of execution time.", "body_md": "i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token\nH100, tg128, greedy: ternary 85.5 >> 151 tok/s. 1-bit 90.1 >> 159 tok/s. logit parity with the fork on 32 of 32 test prompts, gated before any speed number counts. not a lossy trick\nwhy it's faster: at batch-1 the GPU isn't math-bound or bandwidth-bound, it's overhead-bound. the stock path executes 3703 GPU ops per token and spends ~97% of its time on that op overhead. bonsai-turbo fuses the whole per-token pass into a handful of large ops. --mega mode compiles the entire 64-layer token step -- embed >> layers >> logits >> next token -- into one cooperative kernel\nthe kernels were generated by our internal agent -- the same agent that powers @runinfrai\nscope is deliberate: batch-1 decode only. Bonsai 27B is the ternary 27B small enough to run on a phone, and this makes the local single-user experience actually fast. it is not a batched-serving engine\nroofline says ~440-490 tok/s is on the table. next: cp.async weight pipelining, then a speculative drafter -- targeting ~300\ngithub.com/RightNow-AI/bo…", "url": "https://wpnews.pro/news/fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel", "canonical_source": "https://twitter.com/Akashi203/status/2077552491567157733", "published_at": "2026-07-16 00:39:18+00:00", "updated_at": "2026-07-16 00:55:02.602818+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["Infra AI", "PrismML", "Bonsai 27B", "llama.cpp", "H100", "bonsai-turbo", "RightNow AI"], "alternates": {"html": "https://wpnews.pro/news/fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel", "markdown": "https://wpnews.pro/news/fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel.md", "text": "https://wpnews.pro/news/fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel.txt", "jsonld": "https://wpnews.pro/news/fusing-a-27b-ternary-llm-s-whole-decode-step-into-one-cuda-kernel.jsonld"}}