# Run a 35B AI Model on a 24GB Mac: Two New Ways Compared > Source: > Published: 2026-09-17 00:00:00+00:00 A 35B-parameter model at 4-bit precision is a 19.5 GB file. A 24 GB Mac has to hold that file, plus the operating system, plus the working memory the model needs for your conversation, and it cannot. Two releases in the same week, from two different teams, take opposite routes around that wall. Edge0, a paper and open-source framework posted to arXiv on September 16, 2026, leaves the file on the SSD and reads only the parts each word needs. PrismML's Bonsai 2 27B, released September 17, makes the file small enough to fit: 5.9 GB for a 27B model. This post is for a founder or developer who wants a capable model on their own hardware, for privacy or for cost, and needs to know which route fits their machine and their work. Every figure below is the authors' or the vendor's own measurement, labelled as such, from the [Edge0 paper](https://arxiv.org/abs/2609.18063), its [model card](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) and [PrismML's announcement](https://prismml.com/news/bonsai-2-27b), all read September 18, 2026. Nothing here has been reproduced by us or, as far as we can find, by anyone else yet. 1. 01Edge0: 35B on a 24 GB Mac mini at about 20 tokens a second.The paper reports 20.4 tokens a second decode inside 2.9 GiB of peak active memory for a 4-bit Qwen3.6-35B-A3B, with the 19.5 GB checkpoint staying on the SSD. Authors' measurement, one machine. 2. 02Bonsai 2: a 27B model in 5.9 GB, 98.2% of its score.PrismML's ternary version of Qwen3.8 27B is 1.76 bits per weight and, by the vendor's own benchmark suite, keeps 98.2% of the full-precision aggregate. Vendor-run figures. 3. 03They solve different shapes of model.Edge0 works because a mixture-of-experts model uses only a few experts per token, so most weights can stay on disk. Bonsai 2 is a dense model shrunk so all of it fits in RAM. 4. 04Both are Apache 2.0; both are new.Edge0's card says preview. Bonsai 2's whitepaper is on GitHub. Neither has independent benchmarks yet. Test on your own prompts before you build on either. ## 01 — The problemThe memory wall A language model's weights have to be somewhere the processor can reach quickly while it generates each word. On a Mac that is unified memory; on a PC it is the graphics card's memory. If the weights do not fit, the usual answer is a smaller model or a lower-precision copy of the same one. Our guide to [how much memory a model needs](https://www.digitalapplied.com/blog/how-much-vram-run-llm-quantization-kv-cache-context-2026) walks through the arithmetic, and it is unforgiving: memory, not compute, is what stops a consumer machine running a 30B-class model well. The Edge0 paper puts the point sharply for mixture-of-experts models, the design most new open models use. Such a model is built from many small "expert" networks and uses only a few per token. That cuts the compute per token but not the bytes that must be held, and the paper's example is the one in the title: the fully loaded 4-bit weights of its 35B model occupy 18.2 GiB of a 24 GB Mac mini and decode at 3.9 tokens a second, because almost nothing is left for the conversation's working memory. The question both releases answer is how to stop paying for weights you are not using at that moment. ## 02 — Route oneStream experts from the SSD Edge0, by Yu Lin, Yiming Wang, Runyuan Cai, Hanze Liu and Xiaodong Zeng, keeps the whole 4-bit checkpoint on disk, maps it into memory, and reads only the experts the model routes each token to. The obstacle is timing. A mixture-of-experts model decides which experts layer N+1 needs only after layer N has finished, so the disk read cannot start early enough to hide behind the compute. The paper's answer is a "prerouter": a small trained head in each layer that predicts the next layer's routing one token ahead, and that prediction is then used as the routing itself, so the experts staged from disk are exactly the experts used. A second trained piece, a recovery LoRA distilled from the full-precision model, wins back most of the quality lost to 4-bit weights and to replacing the original router. The authors' headline measurement, on a Mac mini M4 Pro with 24 GB, is 20.4 tokens a second of decode inside 2.9 GiB of peak active memory for the 35B tier, with prompt processing at 113 tokens a second cold and 140 warm on a 3,100-token prompt. An 8B tier on the same framework, based on a different base model, runs at 28 tokens a second in 1.5 GiB. The model card's highlights state 15 tokens a second and under 3 GiB, and its performance table gives 14.9 to 17.7 tokens a second on the same Mac mini. Quality is the authors' own OpenCompass run: the 35B tier averages 79.2 across five benchmarks against 83.2 for its fp16 base, a mean gap of 3.9 points; the card describes the pipeline as recovering most of the loss and the paper as leaving the tier within a few points of its teacher. Two figures need their setting. The card's "up to +59%" decode gain from the prerouter is stated without its test machine. The paper's own prerouter comparison runs on a 16 GB MacBook M2 serving an 18.4 GiB checkpoint, where every step must fault experts in from the SSD, and there it reports gains of 80% to 84% depending on how many experts are routed; on the 24 GB Mac mini, the paper reports that disabling the prerouter changes decode from 20.4 to 19.9 tokens a second. And the paper states that single-shot benchmarks on this hardware carry about ±40% run-to-run spread, which is why it reports medians over paired runs. The card's phrase "phone-class memory" describes the memory budget; neither source reports the model running on a phone. ## 03 — Route twoShrink every weight to three values PrismML's route is compression. Ternary Bonsai 2 27B stores each weight as one of three values, −1, 0 or +1, with a 16-bit scale per group, which the company states as 1.76 effective bits per weight and a 5.9 GB total footprint, "more than 9x smaller" than the full-precision Qwen3.8 27B it is built from. Because the model is dense, not mixture-of-experts, all of it is used for every token and all of it has to be in memory; the trick is that 5.9 GB fits almost anywhere. The model supports a 262K-token context and text-and-image input, and is released under Apache 2.0. The quality claim is the vendor's own benchmark suite: an aggregate of 83.9 against 85.4 for the full-precision model, which PrismML reports as 98.2% retention, with the per-category table in its post and full results in a whitepaper hosted on GitHub. The company describes this as making the release practically "lossless"; that is PrismML's word, and the table shows the gaps are smallest on maths and coding, with instruction following slightly above the base model, and largest on vision and knowledge. Speed, also vendor-run: up to 143 tokens a second on an RTX 5090 and 46.8 on an M5 Max, with a stated energy figure of 0.714 mWh per token on an RTX 4090. PrismML says the model runs on Nvidia GPUs through CUDA and on Mac, iPhone and iPad through MLX with custom kernels. We have not seen an iPhone speed figure. Both models descend from Qwen: Edge0's 35B tier from Qwen3.6-35B-A3B, a mixture-of-experts model with about 3B parameters active per token, and Bonsai 2 from Qwen3.8 27B, a dense model one generation newer. A "35B" that activates 3B per token and a "27B" that activates all 27B are not the same amount of model per word, which is one reason their speed figures are not comparable head to head. ## 04 — ComparisonSide by *side* Each cell is what its source states; where the two sources measure different things, the cell says so rather than forcing a match. | Spec | Edge0 35B tier | Bonsai 2 27B | |---|---|---| | Base model | Qwen3.6-35B-A3B, mixture of experts, 256 experts, 4 active per token | Qwen3.8 27B, dense | | What stays in RAM | Only routed experts and the decoder's expert cache: 2.9 GiB peak active (paper); the 19.5 GB checkpoint stays on SSD | The whole model: 5.9 GB | | Speed, as stated | 20.4 tok/s decode, Mac mini M4 Pro 24 GB (paper); card says 15 tok/s | Up to 143 tok/s on RTX 5090; 46.8 tok/s on M5 Max (vendor) | | Quality loss, as stated | Mean 3.9 points below fp16 base across five benchmarks (authors' OpenCompass run) | 98.2% of full-precision aggregate, 83.9 vs 85.4 (vendor suite) | | Hardware it needs | Apple silicon with a fast SSD; framework is MLX-backed. Paper tested 24 GB and 16 GB Macs | Nvidia GPU via CUDA, or Mac, iPhone and iPad via MLX (vendor statement) | | Licence | Apache 2.0 (model card and paper) | Apache 2.0 (vendor post) | | Maturity | Card marked preview; paper v2 dated September 17, 2026; framework, checkpoints and adapters open | Second Bonsai 27B generation; whitepaper on GitHub; weights available | The two speed rows are the ones most likely to be quoted against each other, and they should not be. Edge0's number is a 3B-active model streaming from disk on a mid-range Mac. Bonsai's is a 27B dense model fully in memory on a top-end GPU or Apple's newest laptop chip. The fair comparison is each against the alternative on the same machine: for Edge0 that is the same weights fully resident at 3.9 tokens a second, and for Bonsai it is the full-precision model that would not fit at all. ## 05 — DecisionWhich reader picks which If you are still choosing the machine, our [hardware guide by price bracket](https://www.digitalapplied.com/blog/best-hardware-run-local-ai-models-2026-price-brackets-guide) is written for exactly this decision, and the general trade-offs of 4-bit and lower precision are in [our quantization reference](https://www.digitalapplied.com/blog/quantization-tradeoffs-4bit-8bit-fp8-performance-data). ## 06 — Before you buildWhat to test before relying on either 1. **Your own prompts, not the benchmarks.** Both quality claims are the authors' runs on public benchmarks. Take twenty prompts from the work you actually want the model to do, run them against the full-precision model through an API, and score the differences by hand. 2. **Sustained speed, not first-minute speed.** For Edge0, decode depends on the SSD and on the expert cache staying warm; the paper reports that a cold prompt fills at 113 tokens a second against 140 warm. Run a long session and watch whether the rate holds. 3. **Memory under a real context.** Edge0's 2.9 GiB is the peak at short contexts, as the paper's table note states. Load a 30,000-token document and measure again. Bonsai's 5.9 GB is the weights; the context adds to it. 4. **Thermal and battery behaviour on a laptop.** Streaming from SSD is disk work; ternary decode is compute work. Neither source reports laptop battery figures beyond PrismML's desktop-GPU energy number. 5. **Reproducibility.** Edge0's paper warns of ±40% run-to-run spread on single-shot measurements. Take medians, as the authors did, before you write a number down. For a business, the reason to do this work is data that cannot leave the building or a token bill that has stopped being small. Both routes make a serious model local on hardware you already own. If you want that set up and measured for your own workload, our [AI transformation service](https://www.digitalapplied.com/services/ai-transformation) includes local-model evaluation. ## 07 — Next stepTwo routes, one wall, no winner yet ### Pick the route that matches your model shape, then measure it yourself A mixture-of-experts model on a Mac with a fast SSD points to Edge0; a dense model that must fit in memory, or any Nvidia machine, points to Bonsai 2. Either way, treat the published figures as the authors' claims about their hardware, run twenty of your own prompts against the full-precision model, and log the tokens a second over an hour, not a minute. The releases are days old and both licences let you find out for free.