Ternary Bonsai 2 9B GGUF: Run a 2.4GB Vision Model Locally

Prism ML released Ternary Bonsai 2 9B, a ternary-quantized GGUF build of Qwen3.5-9B that shrinks the vision-language model from 17.9GB to roughly 2.4GB by storing most weights as -1, 0, or +1 with one scale factor per 128-weight group. The model, released under Apache 2.0 with a 262K token context window, runs through llama.cpp and requires a separate mmproj file for image input, with VRAM scalable via the -c context parameter. Testing across coding, reasoning, multilingual, and creative tasks shows a clear quality drop versus the full-precision 9B model, with accuracy falling hardest on rare languages and precise numerical detail.

Ternary Bonsai 2 9B GGUF: Run a 2.4GB Vision Model Locally Prism ML's ternary Bonsai 2 9B shrinks Qwen 3.5 9B to a 2.4GB GGUF file. Here's how it runs on llama.cpp and how well it actually performs. What is Ternary Bonsai 2 9B? Ternary Bonsai 2 9B is a heavily compressed version of Qwen3.5-9B, released by Prism ML in GGUF format for local inference. It shrinks the original 17.9GB model down to roughly 2.4GB by storing most weights as one of three values -1, 0, or +1 instead of standard 16-bit floats. It’s a vision-language model, meaning it accepts both images and text, and it keeps Qwen3.5’s full 262K token context window. The weights are released under Apache 2.0. TL;DR - Ternary quantization replaces each 16-bit weight with one of three values -1, 0, +1 , plus a small per-group scale factor, cutting file size by roughly seven times. - The model shrinks from 17.9GB to about 2.4GB , making a 9-billion-parameter vision-language model runnable on consumer-grade GPUs. - It’s built on Qwen3.5-9B and retains the base model’s 262K token context window , though you can reduce the context length to lower VRAM use. - Hugging Face lists several GGUF variants in the repo, including Q2 0, PQ2 0, PTQ1 0, and DFlash2 formats, plus separate mmproj files needed for vision input. - Testing across coding, reasoning, multilingual, and creative tasks shows a clear quality drop compared to the full-precision 9B model, with accuracy falling hardest on rare languages and precise numerical detail. - The model runs through llama.cpp , with VRAM consumption scalable by adjusting the context size -c parameter at launch. - Despite the compression tradeoffs, it still produces working code, legible reasoning chains, and passable multilingual output for major languages, which is notable for a file this small. - ✕a coding agent - ✕no-code - ✕vibe coding - ✕a faster Cursor The one that tells the coding agents what to build. How does ternary quantization shrink a 9B model to 2.4GB? Standard LLM weights are usually stored as 16-bit floating point numbers, each capable of representing a huge range of precise values like 0.82 or -0.07. Ternary quantization throws almost all of that precision away. Instead of 16 bits per weight, each weight is reduced to one of three possible values: -1, 0, or +1. A single scale factor is kept for every group of 128 weights, which helps preserve the rough magnitude of that group even though individual weights lose their fine detail. The math is straightforward: fewer bits per weight means a dramatically smaller file. Applied to Qwen3.5-9B, this process takes the original 17.9GB model and produces a GGUF file of about 2.4GB, a reduction of roughly seven times. The Hugging Face repository for the model lists multiple quantization variants beyond the base ternary format, including Q2 0, PQ2 0 with and without an MTP component , PTQ1 0, and a DFlash2 build, each representing a different tradeoff between size, speed, and accuracy. Vision support comes through separate mmproj files available in BF16 and Q8 0 , which handle image encoding alongside the main text model. How do you run it locally with llama.cpp? The model ships in GGUF format specifically for llama.cpp, which has built-in support for ternary and other low-bit quantization schemes. To run it as a vision-language model, you need both the main GGUF weights file and the corresponding mmproj file from the same repository, since the projector handles image inputs separately from the language model. Once loaded, llama.cpp exposes the context length through the -c parameter. Since Qwen3.5-9B supports up to 262K tokens of context, running the model at full context length will consume significantly more VRAM than a shorter context window. If you don’t need the full context, reducing this value is the simplest way to cut memory usage, which matters if you’re running on a smaller consumer GPU rather than a datacenter card. The model can run on CUDA or Metal backends, according to the repository’s listed tags, making it usable on both Nvidia GPUs and Apple Silicon. Is Ternary Bonsai 2 9B actually usable, or just a size trick? It depends heavily on the task. In hands-on testing covered by AI engineer Fahad Mirza, the model produced working code: given a photo of a donair kebab and a prompt, it reasoned through the image and generated animation code that rendered a spinning skewer in a browser. That’s a legitimate demonstration that a 2.4GB file can still handle multimodal input and produce coherent, functional output. Remy is new. The platform isn't. Remy is the latest expression of years of platform work. Not a hastily wrapped LLM. Other tests showed more mixed results. Asked to retell a historical event using Nadsat, the invented slang from A Clockwork Orange, the model sprinkled in a handful of correct terms but misused most of the vocabulary, suggesting it leans on surface pattern matching rather than genuine command of the fictional language. A 28-language translation test showed strong results for widely spoken languages like Spanish, Portuguese, and Mandarin, but noticeable degradation in Korean and Indonesian, and outright garbled output in lower-resource languages like Hindi, Urdu, Greek, and several African languages, sometimes with stray words from unrelated languages leaking in. A scientific reasoning test, where the model was asked to critique a wind tunnel infographic like an aerodynamicist, produced a genuinely well-structured reasoning chain and correctly flagged the chart as unreliable it contained AI-generated gibberish formulas and conflicting Reynolds numbers . But the model also invented numbers that weren’t in the image and made arithmetic errors in its own physics checks, a reminder that compressed models can reach the right high-level conclusion while still fabricating specific details. A creative writing test involving a culturally-mixed comedic scene showed strong image description accuracy, but the actual story lost track of plot details the prompt had specified, and the generated dialogue in a second language was weaker than the English portions. What are the tradeoffs compared to the full Qwen3.5-9B? The core tradeoff is precision for portability. The original Qwen3.5-9B model, at nearly 18GB, carries the full expressiveness of 16-bit weights and reflects whatever benchmark performance Qwen’s team achieved at that precision. Compressing to ternary values necessarily discards a lot of that nuance. Based on testing, the practical consequences show up most in: Numerical precision and hallucination: the model can reason its way to a correct overall judgment but still invents specific figures that aren’t grounded in the source material. Low-resource language quality: performance degrades sharply as a language becomes rarer in training data, collapsing into repeated or nonsensical text for some languages tested. Narrative coherence: the model can describe a scene accurately but loses track of plot logic, character details, or specific instructions over the course of generating longer creative text. What it retains reasonably well is image understanding, code generation for well-specified tasks, and reasoning chains that are legible and often land on sensible conclusions, even when the supporting detail is shaky. For anyone needing a lightweight local vision-language model for prototyping, quick image description, or simple coding tasks, that combination can be good enough. For tasks requiring precise numbers, rare-language fluency, or long-form creative consistency, the compression costs become more apparent. Frequently Asked Questions What base model is Ternary Bonsai 2 9B built on? It’s built on Qwen3.5-9B, a 9-billion-parameter vision-language model from Alibaba’s Qwen team. Prism ML applied ternary quantization to compress it, keeping the underlying architecture and the 262K token context window intact. How much VRAM does Ternary Bonsai 2 9B need? VRAM usage scales with context length. Running the full 262K token context window requires substantially more memory than a shorter context. Reducing the -c parameter in llama.cpp lowers VRAM consumption proportionally, so the model can be tuned to fit smaller GPUs by sacrificing maximum context size. What does “ternary” mean in ternary quantization? It means each weight in the model is restricted to one of three values: -1, 0, or +1, instead of a 16-bit floating point number. A separate scale factor is stored per group of weights 128 weights per group in this model to preserve relative magnitude, which keeps the compressed model functional despite the extreme reduction in per-weight precision. Plans first. Then code. Remy writes the spec, manages the build, and ships the app. Is the model good at coding and reasoning? It produced working animation code from an image-and-text prompt and generated coherent, well-structured reasoning chains in testing, including correctly identifying a chart as unreliable gibberish. However, it also made arithmetic errors and invented specific numbers not present in source material, so outputs requiring precise figures should be double-checked. Can I use Ternary Bonsai 2 9B commercially? The model weights are released under the Apache 2.0 license, which generally permits commercial use, modification, and redistribution. Check the specific license file in the Hugging Face repository for exact terms before deploying it in a commercial product.