# Bonsai 2 27B Benchmarks: Does Ternary Quantization Actually Hold Up? > Source: > Published: 2026-09-23 00:00:00+00:00 # Bonsai 2 27B Benchmarks: Does Ternary Quantization Actually Hold Up? Bonsai 2 27B claims 98.2% of FP16 intelligence at ~1.72 bits per weight. Here's how its benchmarks compare to conventional 2-bit and 4-bit builds. ## Direct Answer Bonsai 2 27B is a ternary-weight quantization of a 27B parameter Qwen3.8-based model that stores weights as {-1, 0, +1} instead of 16-bit floats, cutting the language model from roughly 54 GB down to about 7.67 GB. On a 14-benchmark thinking-mode suite, it scores an average of 84.78, which the model card reports as 98.2% of the FP16 original’s intelligence, well above a conventional 2-bit build of the same base model (72.59) and close to a 4-bit build that takes up three times the disk space. The result matters because standard sub-4-bit quantization usually causes visible reasoning collapse, and this one, according to the published numbers, mostly doesn’t. ## TL;DR - **Ternary quantization** stores each weight as one of three values (-1, 0, +1) rather than the 16+ values used in conventional 2-bit or 4-bit schemes, which theoretically caps information per weight at log2(3) ≈ 1.585 bits. - **The reported benchmark average** , 84.78 across 14 thinking-mode tests, is claimed to retain 98.2% of the FP16 baseline’s score, versus 72.59 for a conventional IQ2_XXS quantization of the same base model. - **Math and coding scores hold up especially well** , with math benchmarks landing within half a point of full precision and coding scores matching the unquantized baseline, while agentic tool-calling drops more noticeably to 74.92. - **A blockwise Hadamard rotation** is folded into the stored weights before ternary assignment, which the card credits with helping preserve accuracy that naive ternary rounding would lose. - **Real-world storage cost is higher than the theoretical 1.72 bits/weight** because deployment formats add per-group scale and bias metadata: 2.25 bits/weight in the MLX package used for these benchmarks, 1.75 in the leaner GGUF PTQ1_0 packing. - **Throughput varies by hardware and packing format** , with no single format winning everywhere: PTQ1_0 wins on memory-bandwidth-limited GPUs like the RTX 4090 and L4, while PQ2_0 wins on cards like the RTX 5090 and H100 where decode is limited by compute overhead rather than memory. - **The model runs interactively on a laptop** , reportedly hitting about 47 tokens/second on an Apple M5 Max, something a 54 GB FP16 model simply cannot do on that hardware at all. - ✕a coding agent - ✕no-code - ✕vibe coding - ✕a faster Cursor The one that tells the coding agents what to build. ## What makes this quantization different from a typical 2-bit build? Most “2-bit” quantized models aren’t actually 2 bits per weight once you count the overhead. The model card points out that a widely-used 2-bit build of the same Qwen3.8-27B base model is really closer to 2.8 bits/weight once its metadata is counted, landing at 9.4 GB. Bonsai 2 takes a different approach: it restricts every weight to exactly three values, {-1, 0, +1}, with one shared 16-bit scale per group of 128 weights. That’s an information-theoretic ceiling of about 1.585 bits per weight for the ternary code itself, plus a small amortized cost for the scale factor, bringing the idealized total to roughly 1.71 to 1.72 bits/weight across the model. The catch is that idealized bit-width and deployed bit-width aren’t the same thing. The format needs a container that inference kernels can actually read efficiently, and every container adds some overhead. The GGUF PTQ1_0 packing, which stores trits densely, lands at 1.75 bits/weight, just above the theoretical floor. A second GGUF packing, PQ2_0, stores each trit in a full 2-bit slot for cheaper unpacking, landing at 2.13 bits/weight. The MLX package, the one most of the reported benchmarks were run on, needs both a scale and a bias per group under MLX’s native grouped low-bit format, pushing it to 2.25 bits/weight. Same underlying ternary values in all three, different storage tax. ## How do the benchmark numbers actually break down? The headline figure is an 84.78 average across 14 thinking-mode benchmarks, compared against a conventional IQ2_XXS quantization of the same base model at 72.59, and a much larger UD-Q4_K_XL 4-bit build that scores within 0.4 points of Bonsai 2 despite being roughly three times the footprint. The category breakdown in the model card highlights where the ternary approach holds up and where it doesn’t: - **Math** : 96.57, within half a point of full FP16 precision. This is the standout category. - **Coding** : 89.42, described as level with the FP16 baseline. - **Agentic tool calling** : 74.92, the weakest of the highlighted categories, suggesting multi-step tool-use reasoning is more sensitive to the precision loss than closed-form math or code generation. The pattern across these three numbers suggests ternary quantization degrades unevenly. Tasks with tight, well-defined reasoning chains (arithmetic, code syntax and logic) seem to survive extreme compression better than tasks requiring longer decision chains with more room for small errors to compound, like agentic tool calling. ## Why does a Hadamard rotation matter for a 3-value weight format? Forcing a weight matrix into just three discrete values is a blunt operation. Naive rounding to the nearest of {-1, 0, +1} throws away a lot of the original distribution’s shape, and outlier weights (the few unusually large or small values in a matrix) get hit especially hard, since they get squashed to the same three buckets as everything else. ## Remy doesn't build the plumbing. It inherits it. Other agents wire up auth, databases, models, and integrations from scratch every time you ask them to build something. Remy ships with all of it from MindStudio — so every cycle goes into the app you actually want. Bonsai 2 addresses this with a blockwise Hadamard rotation applied to each weight matrix (in blocks of 1024) before ternary assignment. A Hadamard transform is an orthogonal rotation, meaning it doesn’t lose information on its own, but it redistributes the “energy” of a matrix’s values more evenly across dimensions. That tends to reduce the impact of outliers, since no single dimension carries a disproportionate share of the signal after rotation. The runtime has to apply the matching inverse transform to activations at inference time, which is why the model requires custom kernels rather than standard MLX or llama.cpp loaders. According to the card, this rotation costs no extra storage since it’s folded into the weights offline, but it does mean the model can’t be run correctly with an off-the-shelf loader; using a generic MLX loader on these weights produces wrong output silently, not an error. ## How fast does it run, and does the hardware choice matter? Throughput depends heavily on both the GPU and which GGUF packing is used, and the two packings trade places depending on what’s bottlenecking the hardware. On memory-bandwidth-constrained cards like the RTX 4090, RTX 6000 Ada, L40S, and L4, the leaner PTQ1_0 packing wins on token generation because it moves 17% less data per step. On cards where batch-1 decode is limited by instruction throughput and kernel launch overhead instead, like the RTX 5090, H100 SXM, and RTX PRO 6000 Blackwell, PQ2_0 wins because unpacking dense trits carries its own arithmetic cost that outweighs the bandwidth savings. Prompt processing, being compute-bound rather than memory-bound, favors PQ2_0 across the board. On Apple hardware, the practical story is less about which packing wins and more about the fact that the model runs at all. A 54 GB FP16 model doesn’t fit in unified memory on a typical laptop. The compressed model reportedly streams around 204 GB/s of weight data on an Apple M5 Pro, consistent with a memory-bandwidth-bound decode profile, at 27.5 W on the GPU rail alone. On an M5 Max, the card reports roughly 47 tokens/second for token generation, which is usable for interactive chat on a laptop. ## Is the benchmark comparison fair? The comparisons in the model card are against other quantizations of the same base model (Qwen3.8-27B), rather than against unrelated architectures, which is the right way to isolate the effect of the quantization method itself rather than differences in training or architecture. The FP16 baseline, the conventional IQ2_XXS 2-bit build, and the UD-Q4_K_XL 4-bit build are all derived from the same underlying weights, so the benchmark gaps should reflect quantization quality rather than confounding variables. That said, these are the vendor’s own published numbers rather than independently reproduced third-party results, so treat the specific point values as claims from the model’s authors until independently verified. ## Frequently Asked Questions ### What does “1.72 bits per weight” actually mean here? It’s the idealized, information-theoretic storage cost of the ternary representation: each weight is one of three values (-1, 0, +1), which carries about 1.585 bits of information, plus a small amortized cost for a shared scale factor across each group of 128 weights. Actual deployed formats (GGUF PTQ1_0 at 1.75, GGUF PQ2_0 at 2.13, MLX at 2.25 bits/weight) run slightly higher due to container overhead. ### Why is the MLX version larger than the GGUF versions if they’re the same weights? MLX’s native low-bit format stores both a scale and a bias per group of weights, while the ternary values only need a single scale to be reconstructed exactly. That extra bias field adds no new information but does add storage, pushing the MLX packing to 2.25 bits/weight versus 1.75 for the leaner GGUF PTQ1_0 packing of the identical ternary values. ### Can I run Bonsai 2 27B with standard MLX or llama.cpp loaders? No. The model requires custom kernels because of the Hadamard rotation applied to the weights, which needs a matching inverse transform applied to activations at inference time. A standard loader will load the file without erroring but will produce incorrect output, since it skips the required transform. ### Does ternary quantization fail on any task categories? Based on the published category scores, agentic tool calling shows the largest drop relative to the other highlighted categories, scoring 74.92 against near-parity results in math and coding. This suggests longer, multi-step reasoning chains are more vulnerable to precision loss than shorter, well-defined tasks. ### How does this compare to a standard 4-bit quantization? The reported average benchmark score, 84.78, comes within 0.4 points of a 4-bit (UD-Q4_K_XL) quantization of the same base model, despite the ternary build using roughly a third of the disk space, according to the figures in the model card.