ternary 20B parameter Mixture-of-Experts model — most of those weights are literally baked into 2-bit values, which is why an iPhone can chew through 120 tokens per second without melting.
What's Real Here #
Maple-Preview isn't some benchmark stunt. It's a trimmed-down demo of what the full Maple stack aims to ship: a 20B MoE transformer quantized down to ternary activations and weights, then compiled with a custom backend that leans hard on Apple's Neural Engine. The trick is that only a fraction of experts fire per token, and the ternary representation cuts memory bandwidth demands so drastically that the ANE doesn't become the bottleneck.
The Caveats (Because I'm Skeptical Too) #
The model is, unsurprisingly, dumb as a rock compared to its dense counterparts. You're trading accuracy for sheer speed — think basic instruction following, not nuanced reasoning. The demo also caps sequence length aggressively, so it's fine for quick prompts but falls apart on anything requiring long context. This is very much a proof-of-concept: here's the raw performance, now go build something useful on top.
Setup Notes for Hackers #
The repo ships with a Swift package wrapper around a quantized Core ML model. Running it locally requires:
xcodebuild -project MaplePreview.xcodeproj -scheme MaplePreview -destination 'platform=iOS Simulator,name=iPhone 15'
Memory footprint sits around 400MB once loaded, which is honestly impressive given the parameter count. The tokenizer is a stripped BPE variant, and the MoE routing is handled by a lightweight learned gate that runs entirely on-device.
Why This Matters #
Most "LLM on mobile" demos either punt to the cloud or squeeze a tiny 1B model onto device. Maple-Preview flips that script — it keeps the parameter count high but slashes the computational cost through quantization and expert sparsity. If the full release holds up, we might finally get local LLMs that feel fast instead of tolerating sluggish inference.
The real test will be whether the accuracy cliff is survivable for niche use cases like coding assistants or local knowledge bases. For now, the 120 tok/s number is legit, and that alone makes this worth a look.
OpenAI Just Dragged Apple Into the Dirt Over Those Trade Secret 7h ago
Maple-Preview: 20B MoE Hits 120 tok/s on iPhone 10h ago
Apple's latest legal filing reveals a concerning pattern 18h ago
OpenAI Fires Back at Apple's Trade Secret Lawsuit with Public 18h ago
TurboFieldfare: Running Gemma 4 26B on 8GB Macs 6d ago
Next AI Search Is Weaponizing Reddit Comments Into Stealth Ads →