Bonsai 27B Puts Real Agents on Phones

PrismML released Bonsai 27B, a 1-bit 27B parameter AI model that fits in 3.9 GB, enabling on-device agent loops on iPhones for the first time. The model retains 90% of full-precision performance while running at up to 163 tokens per second on an RTX 5090, bypassing cloud API calls for reasoning, tool use, and vision tasks.

AI https://sourcefeed.dev/c/ai Article Bonsai 27B Puts Real Agents on Phones A 3.9 GB 1-bit 27B model clears the iPhone memory gate that blocked every prior build of this size. Priya Nair https://sourcefeed.dev/u/priya nair A 27B model used to be a laptop-or-server problem. Even a solid 4-bit build lands around 18 GB, which is too big for phones and uncomfortable on most notebooks once you add KV cache and activations. PrismML https://prismml.com/news/bonsai-27b just shipped Bonsai 27B https://github.com/PrismML-Eng/Bonsai-demo , based on Qwen3.6 27B, and the 1-bit variant packs to 3.9 GB. That is small enough to fit an iPhone 17 Pro’s practical app budget, with room left for cache and activations. The ternary sibling is 5.9 GB and aimed at everyday laptops. This is not another “runs on mobile” demo that collapses on multi-step work. The claim worth testing is narrower and more useful: 27B-class reasoning, tool calls, and vision can now live inside the product for agent loops that would otherwise hammer a cloud API a hundred times per task. 1-bit is not just aggressive Q4 Standard quantization starts from a full-precision checkpoint and compresses weights. Bonsai’s 1-bit and ternary models are built for the low-bit regime end to end. Weights sit in {−1, +1} 1-bit, 1.125 effective bits with group-wise FP16 scales or {−1, 0, +1} ternary, 1.71 effective bits . The low-bit path covers embeddings, attention, MLPs, and the LM head. There is no FP16 escape hatch for “important” layers. The vision tower ships separately in compact 4-bit form so screenshots, documents, and camera frames work without blowing the text stack’s footprint. That architecture choice is why the size drop is so large relative to conventional 4-bit builds of the same class, and why the quality story has to be measured against the full-precision baseline rather than against a post-hoc Q2 GGUF. Both variants keep a 262K context window and support speculative decoding. Weights are Apache 2.0. Runtime paths are MLX https://ml-explore.github.io/mlx/ on Apple silicon and CUDA on NVIDIA, with custom kernels for the hybrid-attention layout. Formats are landing in mainline llama.cpp as Q1 0 1-bit, merged and Q2 0 ternary, CPU and Metal in, Vulkan in review . What the benchmarks actually protect PrismML’s 15-benchmark suite thinking mode is the right table to read by capability, not by overall average alone. xychart-beta title "Thinking-mode scores vs full-precision Qwen3.6 27B" x-axis "Math", "Coding", "Agentic", "Instr.", "Knowledge", "Vision", "Overall" y-axis "Score" 0 -- 100 bar 95.3, 88.7, 80.0, 78.4, 83.1, 72.6, 85.0 bar 93.4, 86.0, 74.0, 71.8, 77.0, 65.2, 80.5 bar 91.7, 81.9, 66.0, 65.8, 73.4, 59.6, 76.1 Ternary retains about 95% of the full-precision baseline overall 80.5 vs 85.0 . 1-bit retains about 90% 76.1 . Math and coding barely move. Agentic and tool-calling drop more 80.0 → 74.0 ternary, 66.0 1-bit . Instruction following and vision take a similar hit. For agent work that is mostly plan → code → verify, that distribution is workable. For dense multi-tool orchestration where every structured call has to stick, ternary is the safer default and 1-bit needs evaluation on your tool schemas. PrismML also claims the most aggressive conventional low-bit build of the same base model scores worse than 1-bit Bonsai while using roughly 2.5× the memory, and that 1-bit Bonsai’s intelligence density hits 0.53 per GB more than 10× the full-precision baseline, about 2.7× the best available low-bit alternative . Treat vendor density metrics as directional until independent suites land; the raw size-vs-capability trade still holds from the absolute scores and footprints. Throughput is already in the interactive range on current hardware: up to 163 tok/s 1-bit and 134 tok/s ternary on an RTX 5090; up to 87 and 58 tok/s on an M5 Max. Phone fit is the harder gate. A 12 GB iPhone exposes roughly 6 GB to an app. Storage size alone never told that story; 3.9 GB with headroom for KV cache is what finally clears it. What changes in a developer workflow If you already run local models through llama.cpp or MLX, adoption is mostly packaging and eval, not a new stack. Quick path from the official demo repo: git clone https://github.com/PrismML-Eng/Bonsai-demo.git cd Bonsai-demo export BONSAI MODEL=27B default export BONSAI FAMILY=bonsai 1-bit; ternary is default ./setup.sh ./scripts/start llama server.sh chat + vision + tools on :8080 GGUF and MLX builds are published per size 27B, 8B, 4B, 1.7B . The 27B models are the first vision-language members of the family. Tool calling is OpenAI-style tool calls with round-trips, plus MCP servers in the demo UIs. Thinking effort is budgetable per request. That maps cleanly onto existing agent frameworks that already speak OpenAI-compatible endpoints. Practical decision tree: Laptop agent that codes, reasons, and calls tools offline → Ternary 27B 5.9 GB . Quality is close enough on math/coding that you stop apologizing for “local mode.” Phone or tight RAM, privacy-first assistant over local files/screenshots → 1-bit 27B 3.9 GB . Accept the steeper tool-calling drop and test your exact tools. Still need frontier judgment on hard steps → hybrid: local Bonsai for retrieval, drafting, private context, and routine tools; cloud only when the local model’s confidence or schema success rate falls off. What this replaces is not GPT-class APIs for every query. It replaces the default of shipping every intermediate plan, tool result, and private screenshot to a remote endpoint because nothing capable fit on-device. Marginal cost of a 100-step loop goes to near zero. Latency becomes memory-bandwidth bound, not RTT bound. Offline and air-gapped agents stop being demos. Caveats to budget for in production: KV cache still shares the phone’s ~6 GB envelope, so long 262K contexts are a laptop feature first; vision is 4-bit and weaker than the text stack on the published numbers; and 1-bit tool-calling 66.0 on the suite is the number that will make or break MCP-heavy products. Run your own BFCL-style and schema-strict evals before promising “full agent on device.” The hybrid pattern is the real product shape Cloud APIs stay correct for peak capability and burst scale. The structural change is that non-frontier and privacy-sensitive steps no longer have to pay cloud prices or leave the machine. Route routine multi-step work local, escalate the hard tail. That collapses cost-per-task for agentic systems without pretending a 3.9 GB model is a frontier substitute. Earlier Bonsai 8B/4B/1.7B releases already proved 1-bit could be commercially useful at smaller scale. 27B is the tier where multi-step reasoning, structured tools, and computer-use loops stop feeling like toys. If independent benchmarks hold the math/coding retention and the phone memory claims under real KV load, the pareto shift is real: 27B-class work at a footprint smaller than a full-precision 2B model. Ship ternary on the laptop side today for agent prototypes. Treat 1-bit phone builds as production candidates only after tool-calling evals on your schemas pass. The interesting products will not be “ChatGPT offline.” They will be agents that never needed the network for the private middle of the loop. Sources & further reading - Bonsai 27B 1-bit LLM : The First 27B-Class Model to Run on a Phone https://prismml.com/news/bonsai-27b — prismml.com - PrismML releases Bonsai 27B, claiming first major AI model of its size fit for iPhone - 9to5Mac https://9to5mac.com/2026/07/14/prismml-releases-bonsai-27b-claiming-first-major-ai-model-of-its-size-fit-for-iphone/ — 9to5mac.com - GitHub - PrismML-Eng/Bonsai-demo: Bonsai Demo · GitHub https://github.com/PrismML-Eng/Bonsai-demo/ — github.com - Running a 1-Bit LLM on Windows — Bonsai 8B Setup Guide - Technical Scratches https://blog.danmarshall.dev/articles/1bit-llm/ — blog.danmarshall.dev Priya Nair https://sourcefeed.dev/u/priya nair · AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. Discussion 0 No comments yet Be the first to weigh in.