{"slug": "a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory", "title": "A 35B language model running on an iPhone using only 1–2.5 GB of peak memory", "summary": "Edge0-AI released edge0, an open-source streaming MoE inference framework that runs a 35B-parameter language model on Apple Silicon iPhones at roughly 2.9 GB peak active memory for the edge0-35b tier and about 1.0 GB for edge0-8b. The framework combines SSD expert offload, Recover-LoRA adapters, and a trained prerouter that predicts expert routing one step ahead, which the project says lifts decode throughput by up to 59%. Both tiers ship as end-to-end releases on Hugging Face — Edge0/Edge0-35B-A3B-preview (~23 GB) and Edge0/Edge0-8B-A1B-preview (~4.2 GB) — built on Qwen3.5-MoE 35B-A3B and Ling 3.0 bailing hybrid base models, with the MLX backend requiring macOS on M1/M2/M3/M4 and a CUDA backend on the roadmap.", "body_md": "**An open-source streaming MoE inference framework — SSD expert offload + Recover-LoRA + prerouter routing prediction.**\n\nEnglish | [中文](/Edge0-AI/Edge0/blob/main/README_zh.md)\n\n**edge0** is an open-source streaming MoE inference framework. It\ngeneralizes the production-proven recipe — **SSD expert offload +\nRecover-LoRA + prerouter routing prediction** — into an extensible\nframework. The backend is isolated by design: the current MLX backend\nruns on Apple Silicon, and additional platforms (CUDA, …) plug into the\nsame core abstractions.\n\nTwo model tiers ship with the framework. Each tier is an end-to-end release: the released checkpoint, the trained LoRA adapters, and the trained prerouter heads work together as one unit.\n\n| Tier | Released checkpoint | Inference profile | \n|---|---|---|\n| `edge0-35b` | [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) | 4-bit, 40 layers, 256 experts, prerouter K=4 | \n| `edge0-8b` | [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) | 4-bit, 24 layers, 128 experts, prerouter K=8 | \n\nBoth checkpoints are built on open sparse-MoE base models (Qwen3.5-MoE\n35B-A3B and the Ling 3.0 bailing hybrid respectively) and ship with the\nLoRA and prerouter training done for this framework — the adapter files\nare co-located with each checkpoint and load automatically, so\n`edge0 serve <tier>` runs the trained pipeline out of the box.\n\n- **OS / hardware** : the MLX backend runs on macOS with Apple Silicon\n(M1/M2/M3/M4). The CUDA backend is on the roadmap — no other\nplatforms are supported yet.\n- **Python** : 3.10+ (3.12 recommended).\n- **Memory** : ~2.9 GB peak active memory for`edge0-35b` , ~1.0 GB for`edge0-8b` (short contexts; see[Benchmark](#benchmark) ). Add\nheadroom for the OS, tokenizer, and long-context KV growth.\n- **Disk** : the 4-bit checkpoints are ~23 GB (`edge0-35b` ) and ~4.2 GB\n(`edge0-8b` ); expert weights are mmapped and read on demand, they are\nnot loaded into RAM up front.\n\n- **transformers-style usage** :`AutoModel` /`AutoConfig` /`AutoEngine` resolve the tier from the model name;\n- **Backend isolation** : all MLX code lives under`edge0/backends/mlx/` ;\nthe core logic (model specs, prerouter, streaming expert pool, server)\ndepends only on the backend facade (`edge0/backends/base.py` ), so a new\nbackend implements the same facade (`backends/cuda/` is a reserved\nslot) with zero changes to core code;\n- **Adapters as safetensors** : LoRA and prerouter weights are`.safetensors` files with provenance metadata (source, version, owner\nlayers), resolved from the model directory or`artifacts/` ;\n- **Model + adapters in one directory** : a model directory holds both\nthe base checkpoint (`config.json` /`model*.safetensors` / tokenizer)\nand that model's adapters; upgrading adapters swaps adapter\nfiles only — the base stays read-only and is never merged.\n\n- **SSD expert offload** : expert weights are streamed from storage on\ndemand; peak memory is bounded by the active set, not the parameter\ncount.\n- **Prerouter** : a trained head predicts expert routing one step\nahead, so expert loads overlap the forward pass instead of stalling\nit —**up to +59%** decode throughput; the gain grows with storage\nlatency, model size, and routed width*K* .\n- **Recover-LoRA** : the int4 base is frozen and LoRA adapters are\ntrained by distillation from the FP teacher, recovering most of the\nquantization loss at 4-bit (see[Quality](#quality) ).  Adapters stay\nunmerged: one read-only base serves multiple adapter sets.\n\n```\n# Python >= 3.10; the MLX backend requires macOS with Apple Silicon\npython3.12 -m venv .venv && .venv/bin/pip install -e '.[dev,fetch]'\n```\n\nThe two tiers are published on Hugging Face — each repo bundles the\nbase checkpoint and the trained LoRA + prerouter adapters in **one\ndirectory**, so a single download is a ready-to-run model:\n\n- [`Edge0/Edge0-35B-A3B-preview`](https://huggingface.co/Edge0/Edge0-35B-A3B-preview) (~23 GB)\n- [`Edge0/Edge0-8B-A1B-preview`](https://huggingface.co/Edge0/Edge0-8B-A1B-preview) (~4.2 GB)\n\n```\n# with the repo's helper (defaults to the two repos above):\n.venv/bin/python scripts/fetch_models.py --tier edge0-35b --target-dir models\n.venv/bin/python scripts/fetch_models.py --tier edge0-8b --target-dir models\n\n# or directly with the CLI:\n.venv/bin/huggingface-cli download Edge0/Edge0-35B-A3B-preview     --local-dir models/edge0-35b\n.venv/bin/huggingface-cli download Edge0/Edge0-8B-A1B-preview     --local-dir models/edge0-8b\n```\n\nEither way you end up with a directory like:\n\n```\nmodels/edge0-35b/\n├── config.json, model-*.safetensors, tokenizer files   # base checkpoint\n├── lora_edge0_35b.safetensors          # trained LoRA adapters\n└── prerouter_edge0_35b.safetensors     # trained prerouter heads\n```\n\nTier names resolve to local directories via environment variables (where you put the download is up to you):\n\n```\nexport EDGE0_35B_MODEL=$PWD/models/edge0-35b\nexport EDGE0_8B_MODEL=$PWD/models/edge0-8b\n```\n\nOr skip the env vars entirely and pass the directory directly — the\ntier is auto-detected from the checkpoint's `config.json`:\n\n```\nedge0 demo models/edge0-35b\nedge0 serve models/edge0-8b\n# quick demo\nedge0 demo edge0-35b\n\n# serve (OpenAI-compatible /v1/chat/completions)\nedge0 serve edge0-35b\ncurl http://127.0.0.1:8000/v1/chat/completions \\\n  -H 'Content-Type: application/json' \\\n  -d '{\"messages\":[{\"role\":\"user\",\"content\":\"Hello!\"}],\"max_tokens\":32}'\n\n# 5) One-shot chat (pass --max-new to cap length; add --show-thinking to\n#    print the model's reasoning block too)\nedge0 chat edge0-35b --prompt \"Explain streaming inference in one sentence.\"\n```\n\n`python -m edge0 ...` is equivalent to `edge0 ...`.\n\n``` python\nfrom edge0 import AutoEngine\nfrom edge0.server.chat import ChatMessage, ChatRequest, ChatSession\n\nengine = AutoEngine.from_pretrained(\"/path/to/model\")  # tier auto-detected\nreq = ChatRequest(\n    model=engine.name,\n    messages=[ChatMessage(role=\"user\", content=\"Hello!\")],\n    max_tokens=64,\n)\ntokens, meta = ChatSession(engine, req).run()\nprint(engine._tok.decode(tokens))\nengine.close()   # release mmaps / expert cache\n```\n\n`examples/demo.py` is the same minimal walkthrough (`edge0 demo` runs\nthis exact path).\n\n- **Checkpoint** : the original model directory (`config.json` ,`model*.safetensors` , tokenizer).`edge0 serve <dir>` /`AutoEngine.from_pretrained(<dir>)` detect the tier from`config.json` .\n- **Adapters** (LoRA + prerouter, safetensors) are resolved from either\nlocation automatically:\n  - the model directory (recommended): side by side with the base, e.g.\n`lora_edge0_35b.safetensors` +`prerouter_edge0_35b.safetensors` ;\n  - `artifacts/` (repo root, gitignored): convert once from\ntraining-side npz exports via`edge0 convert-adapters --npz-dir ...` .\n- the model directory (recommended): side by side with the base, e.g.\n- The published model repos bundle both the base checkpoint and the\ncurrent default adapter release, so `scripts/fetch_models.py` produces\na ready-to-run model directory.  Check each model's doc page for its\nadapter provenance (training data, owner-layer layout).\n- Both adapters are required for the prerouter + LoRA pipeline; if a\nfile is missing, `edge0` fails with a clear message (or pass`--no-prerouter` /`--no-lora` to run the plain base model).\n\nAll benchmarks were run by us with [OpenCompass](https://github.com/open-compass/opencompass)\nunder identical settings and parameters for both the edge0 models (int4 +\ntrained adapters + prerouter routing) and the original fp16 base models.\nThe loss of the edge0 pipeline is small: **3.9 points on average for\nedge0-35b, 2.8 for edge0-8b** (MMLU-Pro is even above the base). Max 100:\n\n| Benchmark | edge0-35b (int4) | Qwen3.5-MoE 35B-A3B (fp16) | edge0-8b (int4) | Ling 3.0 tiny (fp16) | \n|---|---|---|---|---|\n| AIME 2026 | 86.6 | 92.7 | 63.3 | 73.3 | \n| HumanEval | 90.9 | 95.1 | 91.5 | 92.7 | \n| GPQA-Diamond | 79.8 | 81.8 | 70.7 | 71.2 | \n| MMLU-Pro | 81.0 | 84.6 | 70.1 | 65.8 | \n| IFBench | 57.9 | 61.7 | 53.9 | 60.6 | \n| **Average** | **79.2** | **83.2** | **69.9** | **72.7** | \n\nMeasured with `examples/bench.py` (3.3k-token prompt prefill → 10 sampled\nwarmup steps → 200 timed sampled decode tokens, 2 runs per tier):\n\n| Tier | Decode speed | Prefill throughput (cold / warm)* | Peak active memory** | Test machine | \n|---|---|---|---|---|\n| `edge0-35b` | 14.9–17.7 tok/s | 113 / 140 tok/s | 2.9 GiB | Mac mini M4 Pro, 24 GB | \n| `edge0-8b` | 23.9–25.3 tok/s | 500 / 1428 tok/s | 1.0 GiB | Mac mini M4 Pro, 24 GB | \n\n*Cold = first request after process start (expert weights fault in from\nSSD); warm = subsequent requests (page cache resident). Prefill numbers\nare throughput over a ~3.3k-token prompt (`BENCH_LONG=1`).*\n\n**Peak active memory at short contexts (MLX allocator peak; expert weights\nstream from SSD via mmap and are not resident). Long contexts add KV\ncache: ~3.3 GiB on `edge0-8b` at 3.3k tokens.*\n\nReproduce:\n\n```\npython examples/bench.py edge0-35b    # via $EDGE0_35B_MODEL\npython examples/bench.py edge0-8b    # via $EDGE0_8B_MODEL\npytest                 # unit tests (no real weights)\nEDGE0_8B_MODEL=/path/to/edge0-8b pytest -m slow -q\n                        # real-weight generation; missing tiers are skipped\n.venv/bin/python scripts/e2e_smoke.py \\\n  --qwen-dir /path/to/edge0-35b --ling-dir /path/to/edge0-8b\n                        # staged vs exact consistency + generation smoke\nscripts/generate_example.py   # full-pipeline API example\nexamples/demo.py       # minimal API walkthrough\n```\n\nApache-2.0, including vendored third-party code (see [NOTICE](/Edge0-AI/Edge0/blob/main/NOTICE)).", "url": "https://wpnews.pro/news/a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory", "canonical_source": "https://github.com/Edge0-AI/edge0/", "published_at": "2026-09-10 15:54:48+00:00", "updated_at": "2026-09-10 16:17:01.044535+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research", "mlops"], "entities": ["Edge0-AI", "edge0", "Edge0/Edge0-35B-A3B-preview", "Edge0/Edge0-8B-A1B-preview", "Qwen3.5-MoE 35B-A3B", "Ling 3.0", "Apple Silicon", "MLX"], "alternates": {"html": "https://wpnews.pro/news/a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory", "markdown": "https://wpnews.pro/news/a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory.md", "text": "https://wpnews.pro/news/a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory.txt", "jsonld": "https://wpnews.pro/news/a-35b-language-model-running-on-an-iphone-using-only-1-2-5-gb-of-peak-memory.jsonld"}}