{"slug": "i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed", "title": "I ran the tiny Bonsai model on my tiny GPU. Here’s how it performed", "summary": "PrismML's 1-bit quantized Bonsai 27B model, weighing 3.9 GB, runs on a consumer GPU with an NVIDIA GeForce RTX 5060, achieving 10-20 tokens per second on average, but slower than smaller models like Qwen 7B. The model supports a 262K token context and speculative decoding, yet tokenization can take over two minutes for large prompts. Testing in LM Studio 0.4.20 on Windows 11 showed that disabling thinking boosts speed but can reduce accuracy, as seen when the model incorrectly attributed 'Blood Meridian' to William Gibson instead of Cormac McCarthy.", "body_md": "The [Bonsai 27B model](https://prismml.com/news/bonsai-27b) from [PrismML](https://prismml.com/) is the latest of many to squeeze a large number of parameters (27 billion) into a footprint compact enough to fit on a smartphone.\n\nThe original model weighed in at 54 GB. However, PrismML claims its 1-bit quantized version — a mere 3.9 GB — can deliver “multi-step reasoning, structured tool calls, vision tasks, and computer-use agentic loops that stay coherent across many steps,” while still retaining much of the accuracy of the larger version, due to “custom low-bit kernels built for its hybrid-attention architecture.”\n\nThe 1-bit quantized version of Bonsai 27B allows for a 262K token context and supports speculative decoding and flash attention. All of this is available with open weights under the Apache 2.0 open-source license.\n\nI gave the 1-bit Bonsai 27B model a spin on my own hardware, using LM Studio 0.4.20 on Windows 11. My PC has an AMD Ryzen 5 3600 six-core CPU (32GB RAM) and an NVIDIA GeForce RTX 5060 (8GB VRAM). I ran Bonsai through what is now my standard test bed for models, a set of prompts covering different tasks:\n\n`pip`\n\ntool has a function `ScriptMaker`\n\n(accessed with `from pip._vendor.distlib.scripts import ScriptMaker`\n\n). On Microsoft Windows this is used to create an .exe stub launcher for a Python package’s entry points when it’s installed with `pip`\n\n. However, the icon created for this stub is the same generic icon used for the Python runtime itself. Let’s write a Python utility to allow the user to append their own custom icon to the .exe stub, but also preserve the stub’s appended archive and other metadata. The utility should use only the Python standard library, and should be kept as simple as possible.”Foundry\n\nBecause Bonsai 27B has such a massive maximum token window (262,144), I set its default token window to 32,767 (up from the usual 4,096 or 16,384 I’ve used for other models). This allowed for complete context injection — for instance, when performing the coding tasks, the entire prompt including the code could be contained comfortably in the token window.\n\nFoundry\n\nThe first thing that became clear: Despite its compactness, Bonsai isn’t the fastest model. One of the factors that impacts token-per-second speeds is the number of parameters in the model, not just the model’s physical size in memory. Bonsai’s 27 billion parameters resulted in token output that was on the whole slower than other models I’ve tested, like the Qwen 7B model. Tokenization also lags. Despite the giant token window I set for Bonsai, I still had to wait upwards of two minutes for my code examples to be tokenized and injected.\n\nThe most I could squeeze out of Bonsai’s output speed was around 40 tokens per second; the average was between 10 and 20. Disabling thinking boosted the output speed, and cut down the time to first token (and the overall output time), but sometimes that came at the cost of accuracy or coherency. For instance, the William Gibson question worked best with thinking on. With thinking off, I got a reply that was outrageously false. (No, William Gibson did not write *Blood Meridian*; Cormac McCarthy did.)\n\nThinking costs also had a major impact in the coding-related prompts. The code modularity question, for instance, took longer than six minutes just in the thinking phase, and the actual response took longer than two minutes to generate at 4.1 tokens per second. With thinking off, the results came more quickly (although there was a fair amount of overhead for tokenizing the input), and Bonsai generated a shorter but still useful reply at 7.7 tokens per second.\n\nOne intriguing capability in Bonsai 27B is its ability to use draft models for [speculative decoding](https://github.com/ggml-org/llama.cpp/blob/master/docs/speculative.md), a technique for speeding up token generation. This it does in place of using multi-token prediction, or MTP. However, speculative decoding requires loading both the main model and the draft model into GPU memory, so it’s only well-suited for cases where you have a lot of VRAM to throw at the problem. (MTP is integrated into the model itself.) My setup did not afford enough memory to use speculative decoding with Bonsai.\n\nBonsai’s strongest features — its giant token window, its massive number of parameters — come at clear costs. Other models with fewer parameters are faster. Set aside speed, though, and Bonsai performs well compared to other models with larger binary footprints. Bonsai’s coding advice and code generation were as good as the output [I got from Gemma 4](https://www.infoworld.com/article/4156597/googles-gemma-4-shines-on-local-systems-both-big-and-small.html) and [from Qwen3.5](https://www.infoworld.com/article/4144487/i-ran-qwen3-5-locally-instead-of-claude-code-heres-what-happened.html). And while I wasn’t thrilled about the tax imposed by enabling thinking, I found myself leaving Bonsai enabled for longer and more complex work. If I’m not getting instant answers, I might as well get well-thought-out ones.\n\nWhat I’m most intrigued by with Bonsai 27B is the underlying quantization techniques used to build the model. Model providers have promised that 1-bit and ternary models can be made to deliver fast and accurate results in a fraction of the space as their higher-quant counterparts. Bonsai shows the promise is viable, and hints that future work in this direction has a lot of potential. For now, though, 1-bit quantized Bonsai 27B is best for those who care more about a small footprint than the fastest output.\n\nIf you want to try Bonsai 27B for yourself, you can [download the model from Hugging Face](https://huggingface.co/collections/prism-ml/bonsai-27b) or [run the model locally in your web browser](https://huggingface.co/spaces/webml-community/bonsai-webgpu-kernels).", "url": "https://wpnews.pro/news/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed", "canonical_source": "https://www.infoworld.com/article/4206771/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed.html", "published_at": "2026-08-17 09:00:00+00:00", "updated_at": "2026-08-17 09:11:57.236586+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-products", "ai-infrastructure"], "entities": ["PrismML", "Bonsai 27B", "LM Studio", "NVIDIA GeForce RTX 5060", "AMD Ryzen 5 3600", "Qwen 7B", "William Gibson", "Cormac McCarthy"], "alternates": {"html": "https://wpnews.pro/news/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed", "markdown": "https://wpnews.pro/news/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed.md", "text": "https://wpnews.pro/news/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed.txt", "jsonld": "https://wpnews.pro/news/i-ran-the-tiny-bonsai-model-on-my-tiny-gpu-heres-how-it-performed.jsonld"}}