A 35B language model running on an iPhone using only 1–2.5 GB of peak memory

Edge0-AI released edge0, an open-source streaming MoE inference framework that runs a 35B-parameter language model on Apple Silicon iPhones at roughly 2.9 GB peak active memory for the edge0-35b tier and about 1.0 GB for edge0-8b. The framework combines SSD expert offload, Recover-LoRA adapters, and a trained prerouter that predicts expert routing one step ahead, which the project says lifts decode throughput by up to 59%. Both tiers ship as end-to-end releases on Hugging Face — Edge0/Edge0-35B-A3B-preview (~23 GB) and Edge0/Edge0-8B-A1B-preview (~4.2 GB) — built on Qwen3.5-MoE 35B-A3B and Ling 3.0 bailing hybrid base models, with the MLX backend requiring macOS on M1/M2/M3/M4 and a CUDA backend on the roadmap.

An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction. English | 中文 /Edge0-AI/Edge0/blob/main/README zh.md edge0 is an open-source streaming MoE inference framework. It generalizes the production-proven recipe — SSD expert offload + Recover-LoRA + prerouter routing prediction — into an extensible framework. The backend is isolated by design: the current MLX backend runs on Apple Silicon, and additional platforms CUDA, … plug into the same core abstractions. Two model tiers ship with the framework. Each tier is an end-to-end release: the released checkpoint, the trained LoRA adapters, and the trained prerouter heads work together as one unit. | Tier | Released checkpoint | Inference profile | |---|---|---| | edge0-35b | Edge0/Edge0-35B-A3B-preview https://huggingface.co/Edge0/Edge0-35B-A3B-preview | 4-bit, 40 layers, 256 experts, prerouter K=4 | | edge0-8b | Edge0/Edge0-8B-A1B-preview https://huggingface.co/Edge0/Edge0-8B-A1B-preview | 4-bit, 24 layers, 128 experts, prerouter K=8 | Both checkpoints are built on open sparse-MoE base models Qwen3.5-MoE 35B-A3B and the Ling 3.0 bailing hybrid respectively and ship with the LoRA and prerouter training done for this framework — the adapter files are co-located with each checkpoint and load automatically, so edge0 serve <tier runs the trained pipeline out of the box. - OS / hardware : the MLX backend runs on macOS with Apple Silicon M1/M2/M3/M4 . The CUDA backend is on the roadmap — no other platforms are supported yet. - Python : 3.10+ 3.12 recommended . - Memory : ~2.9 GB peak active memory for edge0-35b , ~1.0 GB for edge0-8b short contexts; see Benchmark benchmark . Add headroom for the OS, tokenizer, and long-context KV growth. - Disk : the 4-bit checkpoints are ~23 GB edge0-35b and ~4.2 GB edge0-8b ; expert weights are mmapped and read on demand, they are not loaded into RAM up front. - transformers-style usage : AutoModel / AutoConfig / AutoEngine resolve the tier from the model name; - Backend isolation : all MLX code lives under edge0/backends/mlx/ ; the core logic model specs, prerouter, streaming expert pool, server depends only on the backend facade edge0/backends/base.py , so a new backend implements the same facade backends/cuda/ is a reserved slot with zero changes to core code; - Adapters as safetensors : LoRA and prerouter weights are .safetensors files with provenance metadata source, version, owner layers , resolved from the model directory or artifacts/ ; - Model + adapters in one directory : a model directory holds both the base checkpoint config.json / model .safetensors / tokenizer and that model's adapters; upgrading adapters swaps adapter files only — the base stays read-only and is never merged. - SSD expert offload : expert weights are streamed from storage on demand; peak memory is bounded by the active set, not the parameter count. - Prerouter : a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput; the gain grows with storage latency, model size, and routed width K . - Recover-LoRA : the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit see Quality quality . Adapters stay unmerged: one read-only base serves multiple adapter sets. Python = 3.10; the MLX backend requires macOS with Apple Silicon python3.12 -m venv .venv && .venv/bin/pip install -e '. dev,fetch ' The two tiers are published on Hugging Face — each repo bundles the base checkpoint and the trained LoRA + prerouter adapters in one directory , so a single download is a ready-to-run model: - Edge0/Edge0-35B-A3B-preview https://huggingface.co/Edge0/Edge0-35B-A3B-preview ~23 GB - Edge0/Edge0-8B-A1B-preview https://huggingface.co/Edge0/Edge0-8B-A1B-preview ~4.2 GB with the repo's helper defaults to the two repos above : .venv/bin/python scripts/fetch models.py --tier edge0-35b --target-dir models .venv/bin/python scripts/fetch models.py --tier edge0-8b --target-dir models or directly with the CLI: .venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview --local-dir models/edge0-35b .venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview --local-dir models/edge0-8b Either way you end up with a directory like: models/edge0-35b/ ├── config.json, model- .safetensors, tokenizer files base checkpoint ├── lora edge0 35b.safetensors trained LoRA adapters └── prerouter edge0 35b.safetensors trained prerouter heads Tier names resolve to local directories via environment variables where you put the download is up to you : export EDGE0 35B MODEL=$PWD/models/edge0-35b export EDGE0 8B MODEL=$PWD/models/edge0-8b Or skip the env vars entirely and pass the directory directly — the tier is auto-detected from the checkpoint's config.json : edge0 demo models/edge0-35b edge0 serve models/edge0-8b quick demo edge0 demo edge0-35b serve OpenAI-compatible /v1/chat/completions edge0 serve edge0-35b curl http://127.0.0.1:8000/v1/chat/completions \ -H 'Content-Type: application/json' \ -d '{"messages": {"role":"user","content":"Hello "} ,"max tokens":32}' 5 One-shot chat pass --max-new to cap length; add --show-thinking to print the model's reasoning block too edge0 chat edge0-35b --prompt "Explain streaming inference in one sentence." python -m edge0 ... is equivalent to edge0 ... . python from edge0 import AutoEngine from edge0.server.chat import ChatMessage, ChatRequest, ChatSession engine = AutoEngine.from pretrained "/path/to/model" tier auto-detected req = ChatRequest model=engine.name, messages= ChatMessage role="user", content="Hello " , max tokens=64, tokens, meta = ChatSession engine, req .run print engine. tok.decode tokens engine.close release mmaps / expert cache examples/demo.py is the same minimal walkthrough edge0 demo runs this exact path . - Checkpoint : the original model directory config.json , model .safetensors , tokenizer . edge0 serve <dir / AutoEngine.from pretrained <dir detect the tier from config.json . - Adapters LoRA + prerouter, safetensors are resolved from either location automatically: - the model directory recommended : side by side with the base, e.g. lora edge0 35b.safetensors + prerouter edge0 35b.safetensors ; - artifacts/ repo root, gitignored : convert once from training-side npz exports via edge0 convert-adapters --npz-dir ... . - the model directory recommended : side by side with the base, e.g. - The published model repos bundle both the base checkpoint and the current default adapter release, so scripts/fetch models.py produces a ready-to-run model directory. Check each model's doc page for its adapter provenance training data, owner-layer layout . - Both adapters are required for the prerouter + LoRA pipeline; if a file is missing, edge0 fails with a clear message or pass --no-prerouter / --no-lora to run the plain base model . All benchmarks were run by us with OpenCompass https://github.com/open-compass/opencompass under identical settings and parameters for both the edge0 models int4 + trained adapters + prerouter routing and the original fp16 base models. The loss of the edge0 pipeline is small: 3.9 points on average for edge0-35b, 2.8 for edge0-8b MMLU-Pro is even above the base . Max 100: | Benchmark | edge0-35b int4 | Qwen3.5-MoE 35B-A3B fp16 | edge0-8b int4 | Ling 3.0 tiny fp16 | |---|---|---|---|---| | AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 | | HumanEval | 90.9 | 95.1 | 91.5 | 92.7 | | GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 | | MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 | | IFBench | 57.9 | 61.7 | 53.9 | 60.6 | | Average | 79.2 | 83.2 | 69.9 | 72.7 | Measured with examples/bench.py 3.3k-token prompt prefill → 10 sampled warmup steps → 200 timed sampled decode tokens, 2 runs per tier : | Tier | Decode speed | Prefill throughput cold / warm | Peak active memory | Test machine | |---|---|---|---|---| | edge0-35b | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB | | edge0-8b | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB | Cold = first request after process start expert weights fault in from SSD ; warm = subsequent requests page cache resident . Prefill numbers are throughput over a ~3.3k-token prompt BENCH LONG=1 . Peak active memory at short contexts MLX allocator peak; expert weights stream from SSD via mmap and are not resident . Long contexts add KV cache: ~3.3 GiB on edge0-8b at 3.3k tokens. Reproduce: python examples/bench.py edge0-35b via $EDGE0 35B MODEL python examples/bench.py edge0-8b via $EDGE0 8B MODEL pytest unit tests no real weights EDGE0 8B MODEL=/path/to/edge0-8b pytest -m slow -q real-weight generation; missing tiers are skipped .venv/bin/python scripts/e2e smoke.py \ --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b staged vs exact consistency + generation smoke scripts/generate example.py full-pipeline API example examples/demo.py minimal API walkthrough Apache-2.0, including vendored third-party code see NOTICE /Edge0-AI/Edge0/blob/main/NOTICE .