I spent the weekend with Bonsai-27B, and I think I finally understand what a "27B that runs on one GPU" actually means in practice.
The GGUF quants from prism-ml landed on HF this week, so I pulled the Q4_K_M — about 16GB, fits comfortably on a single RTX 3090 with room for a 16K context window. No sharding, no tensor parallelism, no tricks. Just llama.cpp
and a model file. That alone is worth noting because most models that claim "single GPU" actually mean "single GPU with 48GB" or "single GPU if you quantize to 2-bit and squint." This one genuinely works on consumer hardware.
First thing I noticed: it's fast. On my machine I'm getting ~28 tok/s at 4K context, which is usable for chat and fast enough for batch classification work. At 16K it drops to about 19 tok/s — still fine for interactive use. The MoE architecture (35B total parameters, 3B active per token) means you're effectively running a 3B model's compute budget while keeping a 27B's worth of learned parameters in memory. That trade-off is the whole reason this model exists, and it mostly delivers.
I threw a few real tasks at it. Structured JSON extraction from messy invoices — solid, no hallucinations on field names, handled edge cases like missing fields gracefully. A short RAG pipeline over some internal docs — it retrieved and synthesized cleanly at 8K context, though I noticed it occasionally skipped a relevant chunk when the context got dense. That's a known MoE behavior: the router can get confused when too many similar documents compete for attention in the same window.
Where it really struggled was multi-step reasoning with tool calls. I set up a simple agent loop: fetch data, parse it, make a decision, write output. It'd get the first two steps right, then lose the thread on step three. The tool call format would drift, or it'd repeat a previous step instead of moving forward. That's the MoE sparsity tax: the active parameters per token are only 3B, so deep chains of reasoning hit the same ceiling as any small model. You can't cheat your way past that with architecture tricks. If your task needs more than about three reasoning hops, you want a dense model with more active parameters.
The license is Apache 2.0, which means I can actually deploy this without legal review. That alone puts it ahead of several similarly-sized models I've tested this year. No commercial-use caveats, no "ask us nicely" clauses, no patent retaliation gotchas. Just download, run, ship. For a production deployment that matters more than a few points on a benchmark.
Here's what I'd use it for today: classification pipelines, structured extraction, single-turn RAG, and as a fast draft model for speculative decoding with a larger teacher. What I wouldn't use it for: complex agent loops, multi-hop reasoning, or anything that needs to hold a 50-step plan in its head. Know the ceiling before you build on it.
It's not a replacement for a 70B or a 120B. But it's a genuine option for the 80% of production workloads that don't need that much model — and it'll run on hardware you probably already own. That's the kind of open model release I actually find useful.