Run Bonsai 2 27B Locally on Apple Silicon With MLX

How to run the 27B-parameter Bonsai 2 ternary model on Apple Silicon via MLX at 8.6GB, plus GGUF setup for CUDA and llama.cpp.

What is Bonsai 2 27B and why does it run on a laptop? #

Bonsai 2 27B is a 27-billion-parameter language model, built on the Qwen3.8-27B hybrid-attention architecture, that stores nearly all of its weights as ternary values ({-1, 0, +1}) instead of the usual 16-bit floats. That single change takes a model that would normally need roughly 54GB of memory in FP16 down to 8.60GB on disk (7.67GB for the language model plus 0.92GB for the bundled vision tower), which is why it fits and runs interactively on consumer hardware like an Apple Silicon laptop, rather than requiring a data-center GPU.

TL;DR #

• Bonsai 2 27B ships at 8.60GB total through the MLX 2-bit pack, combining a 7.67GB ternary language model with a 0.92GB unquantized vision tower, versus roughly 54GB for the FP16 original.
• It keeps 98.2% of full-precision quality on a 14-benchmark thinking-mode suite (84.78 average), well ahead of a conventional IQ2_XXS build at 72.59, and close to a 4-bit build three times its size.
• On an Apple M5 Max laptop it generates around 47 tokens per second , with prompt processing around 765 tokens per second, using the llama.cpp Metal backend on the GGUF pack.
• **The model requires a custom ** , not a stock MLX or llama.cpp build, because it uses a Hadamard-rotated weight basis that ordinary quantization s don’t know how to reverse.
• Two GGUF packings are available for CUDA and llama.cpp : PTQ1_0 at 5.95GB (denser trit packing) and PQ2_0 at 7.21GB (2-bit slots, cheaper to unpack), and neither one universally wins on speed.
• The MLX version and GGUF versions decode to the same ternary values ; the size difference between them is purely a container/packing property, not a different level of compression.

#

Plans first. Then code.

Remy writes the spec, manages the build, and ships the app.

How does the ternary weight format actually work? #

Each weight in Bonsai 2 takes one of three values: -1, 0, or +1. A group of 128 weights shares a single FP16 scale factor, so the format is called “ternary g128.” Information-theoretically, a ternary digit carries about 1.585 bits, and once you amortize the shared scale across 128 weights, the effective cost lands around 1.71 bits per weight for the language model, 1.72 bits once a small set of higher-precision tensors (about 0.1% of parameters, covering recurrent state and normalization weights) are counted in.

Before quantization, each weight matrix is passed through a blockwise Hadamard rotation, an orthogonal transform applied in fixed blocks of 1024. This rotation is folded into the stored weights ahead of time, so it adds no extra bits or runtime cost on its own, but the corresponding inverse transform has to be applied to activations at inference time. That’s the detail that makes Bonsai 2 incompatible with generic s: skip the rotation and the model doesn’t error out, it just produces wrong output silently.

The vision tower is not part of this scheme. It’s the stock Qwen3.8-27B tower, carried at full FP16 precision, because the rotation and ternary packing only apply to the language model’s linear projections.

Why does the MLX version take more disk space than GGUF? #

The GGUF PTQ1_0 packing lands close to the theoretical minimum, at 1.75 bits per weight and 5.95GB, because it packs trits densely. The MLX 2-bit pack, at 2.25 bits per weight and 7.67GB for the language model, is larger for a structural reason rather than a quality difference: MLX’s grouped low-bit container stores both a scale and a bias per group of 128 weights, while the ternary values {-s, 0, +s} only need the scale. The bias is set to -s to make the math work out, but it duplicates information the format doesn’t strictly need, costing an extra FP16 value per group. The underlying weights are identical. The model card describes this as verified bit-for-bit against the GGUF bands.

There’s also a middle GGUF option, PQ2_0, which stores each trit in a full 2-bit slot (2.13 bits/weight, 7.21GB) instead of packing them densely. It trades some size for cheaper unpacking arithmetic, which matters on hardware where compute, not memory bandwidth, is the bottleneck.

How do you set up Bonsai 2 27B on Apple Silicon with MLX? #

The MLX pack is distributed through Hugging Face and requires a bundled custom runtime, not the stock MLX , because of the Hadamard rotation. The basic flow:

hf download prism-ml/Ternary-Bonsai-2-27B-mlx-2bit --local-dir bonsai2-27b-mlx
pip install -r bonsai2-27b-mlx/runtime/requirements.txt

Then load it through the runtime module that ships in the pack rather than a generic mlx_vlm model :

import sys
sys.path.insert(0, "bonsai2-27b-mlx/runtime")
from vision_artifact import load_vl_model, chat_config
from mlx_vlm import generate
from mlx_vlm.prompt_utils import apply_chat_template

model, processor, config = load_vl_model("bonsai2-27b-mlx")
prompt = apply_chat_template(processor, chat_config(config), "What is in this image?", num_images=1)
print(generate(model, processor, prompt, ["photo.jpg"], max_tokens=256, temperature=1.0))

Remy is new. The platform isn't. #

Remy is the latest expression of years of platform work. Not a hastily wrapped LLM.

For text-only use, just omit the image argument. The model declares its architecture as prism_hadamard_qwen35 in its config, which is a signal to any that it needs the special activation transform. An ordinary MLX that doesn’t recognize this will load the file and run, but it will not apply the inverse Hadamard transform or the correct embedding lookup, and the output will be wrong without any error being raised. That’s worth flagging clearly: silent wrongness is a worse failure mode than a crash.

What speed can you expect on Apple hardware? #

Reported figures on the llama.cpp Metal backend, using the GGUF packs at a 7.2GB footprint, show a clear scaling pattern across Apple’s laptop chips:

On the M5 Pro specifically, decode throughput streams roughly 204 GB/s of weight data, which lines up with the model’s memory-bandwidth-bound profile: a low-bit model is designed to be small enough that reading the weights, not raw compute, is the limiting factor during generation. Power draw is modest too, with the M5 Pro’s GPU rail measured at 27.5W during decode, versus 300 to 455W of board power for the datacenter GPUs used in cross-platform benchmarking. The tradeoff is that on very long prompts on slower chips like the M4 Pro, prefill speed (prompt processing) becomes the practical bottleneck rather than generation speed.

What about CUDA and llama.cpp on non-Apple hardware? #

The same weights ship as a separate GGUF release for CUDA, CPU, and Metal-via-llama.cpp, in the PTQ1_0 and PQ2_0 packings described above. Running them requires a specific llama.cpp fork built to handle the ternary hybrid-attention kernels; stock llama.cpp builds cannot load these files correctly, similar to the MLX situation.

Across a range of NVIDIA GPUs, neither packing wins consistently. PTQ1_0 (the smaller, denser packing) tends to win on the memory-bandwidth-limited cards, like the RTX 4090, RTX 6000 Ada, and L4, where moving 17% less data per step matters more than the extra unpacking arithmetic. PQ2_0 tends to win on cards like the RTX 5090, H100, and Blackwell-generation parts, where batch-1 decoding is limited by instruction throughput rather than memory bandwidth. Prompt processing, being compute-bound rather than bandwidth-bound, favors PQ2_0 across the board.

Is running Bonsai 2 27B locally actually worth it? #

For anyone who wants a genuinely large, reasoning-capable model running entirely on a laptop, the appeal is straightforward: a 27B model that scores close to its full-precision counterpart on math, coding, and agentic tool-calling benchmarks, fitting in under 9GB of memory, is not something the FP16 version of the same model can offer at all (it simply doesn’t fit). The tradeoff is setup friction. This isn’t a drop-in GGUF you point a standard llama.cpp binary at; it needs matching forks and runtime code because of the rotation scheme. Anyone deploying it should treat the project’s own demo repository as the reference implementation rather than improvising a .

Frequently Asked Questions #

How much RAM does Bonsai 2 27B need to run?

The full pack, including the vision tower, is 8.60GB on disk under the MLX packaging. The GGUF language-model-only packs are smaller: 5.95GB for PTQ1_0 and 7.21GB for PQ2_0. Actual runtime memory will be slightly above the on-disk size to account for context and activations, but it comfortably fits on laptops with 16GB or more of unified memory.

Can I use a standard llama.cpp or MLX build to run it?

Other agents ship a demo. Remy ships an app. #

Real backend. Real database. Real auth. Real plumbing. Remy has it all.

No. The model uses a Hadamard-rotated weight basis (model_type: prism_hadamard_qwen35) that generic s don’t know to reverse. A standard will run without erroring but will produce incorrect output. You need the project’s llama.cpp fork for CUDA/Metal/CPU or the bundled runtime for MLX.

What’s the difference between the PTQ1_0 and PQ2_0 GGUF packs?

PTQ1_0 packs ternary values densely at 1.75 bits per weight (5.95GB) and wins on memory-bandwidth-limited GPUs. PQ2_0 stores each value in a 2-bit slot at 2.13 bits per weight (7.21GB), costing more space but less unpacking arithmetic, which wins on compute-limited GPUs and in prompt processing generally.

How does quality compare to the full-precision model?

On a 14-benchmark thinking-mode suite, Bonsai 2 averages 84.78, described as 98.2% of the FP16 baseline’s intelligence, compared to 72.59 for a conventional IQ2_XXS quantization at a larger footprint. Math, coding, and agentic tool-calling scores are all reported close to full precision.

Does it support vision input?

Yes. The MLX pack bundles the official Qwen3.8-27B vision tower at full FP16 precision (0.92GB), unquantized and untouched by the ternary rotation, so image inputs work alongside the compressed language model.