Edge0 streams MoE experts off SSD to fit 35B in 3 GB

Edge0-AI has open-sourced Edge0, an Apache 2.0 streaming inference engine that memory-maps sparse mixture-of-experts weights from SSD so a 35B model runs in roughly 2.9 GiB of peak active memory. The project ships two 4-bit preview checkpoints, Edge0-35B-A3B (built on Qwen3.5-MoE 35B-A3B) and Edge0-8B-A1B (built on inclusionAI's Ling 3.0 tiny), and reports 14.9-17.7 tok/s decode for the larger model on a 24 GB Mac mini M4 Pro. A developer's analysis notes that at those speeds most expert reads must be served from the OS file cache, meaning machines with less RAM will fall back to the drive.

Where does a 35B model go when it only takes 2.9 GB of RAM? I went into Edge0 to find out, and I came out with a different mental model of what a local model costs. Edge0-AI open-sourced Edge0 the other day: a streaming inference engine under Apache 2.0, plus two preview models built on open sparse MoE bases. Edge0-35B-A3B sits on Qwen3.5-MoE 35B-A3B, 40 layers and 256 experts. Edge0-8B-A1B sits on inclusionAI's Ling 3.0 tiny, a hybrid that pairs multi-head latent attention MLA with MoE, 24 layers and 128 experts, about 1.2B active out of 7.9B. Both ship as 4-bit checkpoints, 19.6 GB and 4.55 GB on disk by the model cards. The clip of a 35B model answering on an iPhone at about 1 GB of peak memory is what made the rounds 🤯 In a mixture-of-experts model most of the parameters live in the experts, and the router only calls a few of them per token: 4 out of 256 here, about 3B active parameters out of 35B. A dense runtime loads every weight because every weight is used for every token. A MoE runtime has a choice, and Edge0 takes the aggressive one. The whole checkpoint stays on the SSD, the expert weights are memory-mapped, and only the routed experts are read as each token comes through. The project's own line is that peak memory is bounded by the active set rather than by the parameter count. The numbers, all from the project and measured on a Mac mini M4 Pro with 24 GB of unified memory: | | edge0-35b | edge0-8b | |---|---|---| | Peak active memory, short context | 2.9 GiB | 1.0 GiB | | Decode | 14.9-17.7 tok/s | 23.9-25.3 tok/s | | Prefill | 113 / 140 tok/s | 500 / 1428 tok/s | | Checkpoint on disk | 19.6 GB | 4.55 GB | The footnote on the memory row matters more than the row. It is a short-context figure: the KV cache sits in RAM, so a long prompt grows it, 3.3 GiB at 3.3k tokens on the 8B by the project's own note. Whatever you plan around, plan around that line moving. Three things I worked out on top of the table. First, 35B at 4 bits is 17.5 GB of weights, so the 19.6 GB checkpoint carries about 2 GB at higher precision, presumably the embeddings, the attention weights and the quantisation scales. Second, 3B active at 4 bits is about 1.5 GB, so roughly half of the 2.9 GB peak is the routed experts and the other half is shared weights plus cache. Third, and this is the one I would watch: at 15 to 18 tokens a second, pulling a full 1.5 GB active set cold off the drive per token would need over 20 GB/s, and no consumer SSD does that. So most expert reads must be served from the operating system's file cache, and my reading of "peak active memory" is that it counts the process's resident set while the mmapped pages the OS keeps cached sit outside it. A 24 GB Mac mini can hide a large share of the 19.6 GB checkpoint in that cache. A Mac with less memory cannot, and that is where the drive starts doing real work. I have not checked this against the code, so treat it as a reading, and as the first thing to measure on your own machine 🧐 Install is a Python package, and the project wants Python 3.10 or newer on an Apple Silicon Mac. The one-liner: pip install -e 'git+https://github.com/Edge0-AI/edge0.git egg=edge0 fetch ' Inside a checkout the project's own line is this, with the fetch extra pulling the checkpoint: python3.12 -m venv .venv && .venv/bin/pip install -e '. dev,fetch ' Budget the 19.6 GB before you start, and put it on the drive you want reading from on every token, because that is what it will do. One thing I hit in the repo rather than on a machine: garbled, mixed-language output on Apple A18 and A18 Pro is listed as a symptom of an older mlx. That is the only chip-level mention of an iPhone anywhere in the repo, and it points at a library version rather than at a build you can install on a phone. RAM is what you save. Disk, cache and speed are what you spend. The 35B checkpoint sits on the SSD for the whole session, decode is 14.9-17.7 tok/s on a desktop-class M4 Pro, and each token is a storage or cache read. I would want to know what that does to a drive over a few months before making it a daily driver, and the project does not say. The phone is in the demo video. The project says the MLX backend runs on macOS with Apple Silicon M1 through M4 and that no other platform is supported yet, CUDA included. The "iPhone 15 Pro and newer" floor circulating in reposts appears nowhere in the README or the model cards, and what you can run today is a Python package on a Mac with 20 GB free. Both models are also previews. The project's own eval has the 35B pipeline at 79.2 against 83.2 for the fp16 base, and tool use, multi-step planning and long-horizon autonomy are called weak. So this is a chat model that fits in a small RAM budget, and an agent backbone it is not yet. Anyone with an Apple Silicon Mac that has more disk than RAM, which is most laptops. The mental model I took away: once expert weights stream from storage per token, the ceiling on a local model is your disk size, and RAM becomes a working-set budget, the way it is for a database. Repo: https://github.com/Edge0-AI/edge0/ https://github.com/Edge0-AI/edge0/ and the 35B card: https://huggingface.co/Edge0/Edge0-35B-A3B-preview https://huggingface.co/Edge0/Edge0-35B-A3B-preview If you run MoE models locally: do you already keep them on SSD with mmap through another runtime, and how did decode hold up past a few thousand tokens of context?