cd /news/artificial-intelligence/fusing-a-27b-ternary-llm-s-whole-dec… · home topics artificial-intelligence article
[ARTICLE · art-61301] src=twitter.com ↗ pub= topic=artificial-intelligence verified=true sentiment=↑ positive

Fusing a 27B ternary LLM's whole decode step into one CUDA kernel

Infra AI open-sourced bonsai-turbo, a batch-1 decode engine that runs PrismML's Bonsai 27B ternary LLM 1.76x faster than the official llama.cpp fork on an H100, achieving 151 tok/s (ternary) and 159 tok/s (1-bit) with exact logit parity. The speedup comes from fusing the entire per-token decode step into a single CUDA kernel, eliminating GPU op overhead that previously consumed 97% of execution time.

read1 min views1 publishedJul 16, 2026
Fusing a 27B ternary LLM's whole decode step into one CUDA kernel
Image: source

i open-sourced bonsai-turbo -- a batch-1 decode engine that runs @PrismML's Bonsai 27B 1.76x faster than the official llama.cpp fork. same outputs, token for token H100, tg128, greedy: ternary 85.5 >> 151 tok/s. 1-bit 90.1 >> 159 tok/s. logit parity with the fork on 32 of 32 test prompts, gated before any speed number counts. not a lossy trick why it's faster: at batch-1 the GPU isn't math-bound or bandwidth-bound, it's overhead-bound. the stock path executes 3703 GPU ops per token and spends ~97% of its time on that op overhead. bonsai-turbo fuses the whole per-token pass into a handful of large ops. --mega mode compiles the entire 64-layer token step -- embed >> layers >> logits >> next token -- into one cooperative kernel the kernels were generated by our internal agent -- the same agent that powers @runinfrai scope is deliberate: batch-1 decode only. Bonsai 27B is the ternary 27B small enough to run on a phone, and this makes the local single-user experience actually fast. it is not a batched-serving engine roofline says ~440-490 tok/s is on the table. next: cp.async weight pipelining, then a speculative drafter -- targeting ~300 github.com/RightNow-AI/bo…

── more in #artificial-intelligence 4 stories · sorted by recency
── more on @infra ai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/fusing-a-27b-ternary…] indexed:0 read:1min 2026-07-16 ·