{"slug": "edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram", "title": "Edge0-35B-A3B: How a 35B MoE Model Runs in 3GB of RAM", "summary": "Edge0 released Edge0-35B-A3B-preview, a 35-billion-parameter mixture-of-experts language model built on Qwen3.6-35B-A3B that runs with under 3 GiB of active memory by streaming experts from SSD instead of loading all weights into RAM. The Apache 2.0 model uses 256 experts with 4 active per token, a prerouter that adds up to 59% more decode throughput, and a Recover-LoRA distillation technique that lands 3.9 points below the fp16 original on average across five benchmarks. On a Mac mini M4 Pro with 24GB, the model decodes at roughly 15 to 18 tokens per second and prefills long prompts at 113 to 140 tokens per second, running on Apple Silicon via MLX as a preview that is weaker at agentic and tool-use tasks.", "body_md": "# Edge0-35B-A3B: How a 35B MoE Model Runs in 3GB of RAM\n\nEdge0-35B-A3B streams MoE experts from SSD to run a 35B-parameter model in under 3GB active memory. Setup, speed, and hardware needs explained.\n\n## What is Edge0-35B-A3B?\n\nEdge0-35B-A3B-preview is a 35-billion-parameter mixture-of-experts (MoE) language model, built on Qwen3.6-35B-A3B, that runs with under 3 GiB of active memory instead of the tens of gigabytes normally required for a model this size. It does this by keeping the full 4-bit weight set on storage and streaming only the experts a given token actually needs, rather than loading the entire model into RAM. The release comes from Edge0, paired with an open-source inference framework of the same name, and is licensed Apache 2.0.\n\n## TL;DR\n\n- **Edge0-35B-A3B** is a 35B-parameter MoE model (256 experts, 4 active per token) that keeps peak active memory around**2.9 to 3 GiB** by streaming experts from SSD instead of loading all weights into RAM.\n- The core trick is **SSD expert offload** : since only 4 of 256 experts fire per token, the framework fetches just those weights on demand, so memory scales with the active set, not the full parameter count.\n- A **prerouter** module predicts which experts will be needed one step ahead, overlapping storage reads with computation and adding up to 59% more decode throughput.\n- A distillation technique called **Recover-LoRA** trains adapters on top of a frozen int4 base to close most of the quality gap that normally comes with 4-bit quantization, landing 3.9 points below the fp16 original on average across five benchmarks.\n- On a **Mac mini M4 Pro (24GB)** , the model decodes at roughly 15 to 18 tokens per second and prefills long prompts at 113 to 140 tokens per second.\n- The current release runs on **Apple Silicon via MLX** ; it’s explicitly a preview, weaker at agentic/tool-use tasks, and other backends are described as being on the roadmap rather than shipped.\n- Setup is a pip install plus a Hugging Face download, then a single CLI command (`edge0 chat` or`edge0 serve` ) to run it locally or expose an OpenAI-compatible API.\n\n## Other agents ship a demo. Remy ships an app.\n\nReal backend. Real database. Real auth. Real plumbing. Remy has it all.\n\n## How does a 35B model fit in 3GB of memory?\n\nThe short answer: it doesn’t, not all at once. A 35B-parameter model at 4-bit quantization would normally need somewhere around 17 to 20GB just to hold the weights, before accounting for activations and KV cache. Edge0-35B-A3B sidesteps this by exploiting how MoE architectures actually compute a forward pass.\n\nIn a sparse MoE model, each token only routes through a small subset of the total experts. Here, the model has 256 experts total but activates just 4 per token (this is the “A3B” in the name, referring to roughly 3B active parameters per forward pass despite the 35B total). Edge0’s framework keeps the full checkpoint on disk (SSD or fast internal flash) and pulls only the weights for the experts actually routed to, on the fly, for each token. Nothing is sharded across devices and nothing forces the full parameter set into memory upfront. Peak memory is bounded by the size of the active expert set plus whatever KV cache the context length demands, not by the 35B total.\n\nThis is why the number that matters most in the model card isn’t parameter count, it’s the measured 2.9 GiB peak active memory on a Mac mini M4 Pro with 24GB of unified memory, run through the project’s own benchmark script.\n\n## What is a “prerouter” and why does it matter for speed?\n\nStreaming weights from storage on every forward pass sounds like it should be slow, and naively it would be: every expert fetch becomes a stall while the system waits on disk I/O. Edge0’s answer to this is a small trained component called a prerouter.\n\nThe prerouter’s job is to predict, one step ahead of time, which experts the next token is likely to need. That prediction lets the framework start fetching those expert weights from storage while the current forward pass is still computing, so the I/O overlaps with compute instead of blocking it. According to the model card, this overlap yields up to a 59% improvement in decode throughput, and the benefit grows as storage gets slower, as the model gets bigger, or as the routed width (how many experts each token uses) increases. In other words, the prerouter is what turns “technically possible” into “usable at interactive speed.”\n\nCombined, the SSD offload and prerouter give the measured performance figures: 14.9 to 17.7 tokens per second on decode, and 113 tokens per second cold / 140 tokens per second warm on prefill, all measured on a Mac mini M4 Pro with 24GB of memory using the project’s `bench.py` script.\n\n## Does quantization to 4-bit hurt output quality?\n\n### Built like a system. Not vibe-coded.\n\nRemy manages the project — every layer architected, not stitched together at the last second.\n\nSomewhat, but less than typical 4-bit quantization would suggest. Edge0 addresses this with a technique it calls Recover-LoRA: the int4 base model is frozen, and LoRA adapters are trained via distillation from the original fp16 model to recover quality lost during quantization. Because the adapters are stored separately rather than merged into the base weights, one read-only int4 base can serve multiple different adapter sets, which also matters for anyone wanting to batch-serve variants without re-quantizing from scratch.\n\nThe model card reports benchmark comparisons run with OpenCompass, using identical settings for both the int4 Edge0 pipeline and the fp16 Qwen3.6-35B-A3B base:\n\n| Benchmark | Edge0-35B (int4) | Qwen3.6-35B-A3B (fp16) | \n|---|---|---|\n| AIME 2026 | 86.6 | 92.7 | \n| HumanEval | 90.9 | 95.1 | \n| GPQA-Diamond | 79.8 | 81.8 | \n| MMLU-Pro | 81.0 | 84.6 | \n| IFBench | 57.9 | 61.7 | \n| Average | 79.2 | 83.2 | \n\nThe gap averages 3.9 points across these five benchmarks, which is a modest loss given the model drops from full precision to 4-bit and also has to stream experts from disk rather than holding them resident.\n\n## What hardware do you actually need to run it?\n\nThe published benchmarks were run on a Mac mini M4 Pro with 24GB of unified memory, and the framework’s backend (MLX) currently targets Apple Silicon specifically. That means, in practice, this is a Mac-first release right now: other backends are listed as being on the roadmap but aren’t available yet.\n\nBeyond the chip architecture, the two things that matter most are storage speed and available RAM headroom. Since experts stream from SSD on every forward pass, fast NVMe or internal flash storage is effectively a requirement, not just a nice-to-have, the whole design assumes storage latency is low enough for the prerouter’s prefetching to hide it. On the memory side, the 2.9 GiB figure applies to short contexts; longer conversations grow the KV cache on top of that baseline, so the model card recommends keeping contexts shorter to stay near the advertised memory ceiling.\n\n## Is Edge0-35B-A3B worth using right now?\n\nIt depends on what you need it for. As a preview release, the model card is upfront about limitations: it’s primarily tuned for the base model’s languages, and it explicitly says agentic capability (tool use, multi-step planning, long-horizon autonomy) is currently weak, with the full release expected to improve this substantially. So if you’re building an agent framework or a tool-calling pipeline today, this preview isn’t the target use case yet.\n\nWhere it looks more immediately useful is on-device chat and reasoning tasks on hardware that couldn’t otherwise touch a 35B-class model: Apple Silicon machines with limited unified memory, or scenarios where you want to batch-serve several LoRA-adapted variants from one base checkpoint without duplicating the quantized weights for each one. The framework and both the 35B and a smaller 8B-A1B variant are released under Apache 2.0, so there’s no licensing friction to experimenting with it.\n\n## Frequently Asked Questions\n\n### How much RAM does Edge0-35B-A3B need to run?\n\nThe measured peak active memory is about 2.9 GiB on a Mac mini M4 Pro, for short contexts. Longer conversations add to this through KV cache growth, so actual memory use will be somewhat higher for extended chats.\n\n### What makes this different from a normal quantized MoE model?\n\nMost quantized models still load the full weight set into memory even at 4-bit. Edge0 instead streams only the experts actually routed to for each token directly from SSD, combined with a prerouter that prefetches likely-needed experts ahead of time to hide storage latency.\n\n### Everyone else built a construction worker.\n\nWe built the contractor.\n\nOne file at a time.\n\nUI, API, database, deploy.\n\n### Can I run Edge0-35B-A3B on Windows or Linux with a GPU?\n\nNot currently. The framework’s backend targets Apple Silicon via MLX. Other backends are mentioned as being on the roadmap but aren’t part of this preview release.\n\n### How does the quality compare to the original Qwen3.6-35B-A3B model?\n\nAcross five OpenCompass benchmarks (AIME 2026, HumanEval, GPQA-Diamond, MMLU-Pro, IFBench), the int4 Edge0 pipeline scores an average of 79.2 versus 83.2 for the fp16 base, a 3.9 point gap attributed to the Recover-LoRA distillation process used to offset quantization loss.\n\n### Is Edge0-35B-A3B good for agentic tasks or tool use?\n\nNot yet. The model card explicitly flags this preview as weak on tool use, multi-step planning, and long-horizon autonomy, and states that a future full release will focus on strengthening these capabilities.", "url": "https://wpnews.pro/news/edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram", "canonical_source": "https://www.mindstudio.ai/blog/edge0-35b-a3b-preview-phone-memory-llm/", "published_at": "2026-09-15 00:00:00+00:00", "updated_at": "2026-09-15 17:18:24.645324+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "machine-learning", "ai-research"], "entities": ["Edge0", "Edge0-35B-A3B-preview", "Qwen3.6-35B-A3B", "Recover-LoRA", "Apple Silicon", "MLX", "Mac mini M4 Pro", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram", "markdown": "https://wpnews.pro/news/edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram.md", "text": "https://wpnews.pro/news/edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram.txt", "jsonld": "https://wpnews.pro/news/edge0-35b-a3b-how-a-35b-moe-model-runs-in-3gb-of-ram.jsonld"}}