{"slug": "make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model", "title": "Make your own Darkbloom – The Expert Network pools Macs into one model", "summary": "A new open-source project, Expert Sniper, pools ordinary Macs into a network that runs Mixture-of-Experts (MoE) models none of them could run alone, achieving 4.0 tok/s on a 16 GB M4 mini running Qwen3-Coder-30B-A3B 4-bit (17 GB on disk) and 19.3 tok/s on a two-node localhost setup with OLMoE-1B-7B. The project exploits MoE sparsity to stream only active experts from SSD or remote nodes, with each node owning a partition of experts via rendezvous hashing, and it is available via pip install with commands like mlx-sniper download and expert-node.", "body_md": "**Connect ordinary computers into a network that runs MoE models none of them\ncould run alone.**\n\nMoE (Mixture-of-Experts) models activate only 8 of 128-256 experts per token,\nso ~97% of the weights are unused per computation. That sparsity has a\nconsequence most inference stacks ignore: the experts don't have to live on\nthe machine doing the thinking. They can live on *any* machine — on its SSD,\nserved over the network — because what moves between boxes is activations\n(~4 KB per layer), while what moves within a box is expert weights\n(~13.5 MB per layer). Interconnect is never the bottleneck; pooled SSD\nbandwidth is the resource, and it scales linearly with every machine that\njoins:\n\n```\nfeasible when   N x SSD_bandwidth x cache_hit_rate  >  bytes_per_token x target_tok/s\n                └ number of machines in the network\n```\n\nExpert Sniper builds this in two tiers:\n\n**One machine**(`src/mlx_expert_sniper/`\n\n): pin the small always-needed weights (attention, norms, router — ~0.9-1.4 GB) in RAM and stream only the active experts from SSD via`F_NOCACHE`\n\n+`pread`\n\n, with a right-sized LRU expert cache and threaded prefetch. A 17-21 GB model runs in a few GB of RAM.**A network of machines**(`src/expert_network/`\n\n): each node owns a partition of the experts, loaded from its own SSD into its own RAM; a driver runs attention locally and dispatches expert compute to whichever nodes own the active experts. Every machine added is more expert RAM and more SSD bandwidth. Partition assignment is rendezvous hashing over a shared node roster — every node derives the same assignment with zero coordination, and adding a node moves only ~1/N of the experts.\n\n**One machine, measured** (16 GB M4 mini, Qwen3-Coder-30B-A3B 4-bit, 17 GB\non disk): **4.0 tok/s, 72.5% expert-cache hit rate, 5.4 GB/s effective\nreads, ~3.6 GB peak RAM.** The model is larger than the machine's memory.\n\n**Two nodes + driver** (localhost, OLMoE-1B-7B, 64 experts split 32/32 by\nroster): **19.3 tok/s, TTFT 0.6 s, 2.9 ms/layer round trip, ~0.24 MB of\nnetwork traffic per token** — faster than the same model streaming from SSD\non one machine (14.9 tok/s), because resident partitions take the SSD out of\nthe token loop entirely. *That measurement is localhost.* Across two physical\nmachines it is a well-founded projection, not a result; see issue #3.\n\n**Joining does not mean downloading the whole model.** A node fetches only\nthe experts it owns, from machines already serving them — the\n`/block/{layer}/{expert}`\n\nendpoint that proves a node holds an expert is the\nsame one that hands it to the next node, and every block is checked against\na published sha256 manifest on arrival. Blocks land in sparse files at the\nformat's fixed offsets, so the engine reads them unchanged while the disk\nholds only this machine's slice (measured: 0.2 GB on disk for a partition of\na 4 GB model).\n\n```\npip install -e \".[network]\"\nmlx-sniper download olmoe-1b-7b -o ~/models/olmoe-stream   # 3.6 GB demo model\n\n# each machine (or terminal) derives its partition from the shared roster:\nexpert-node --model-dir ~/models/olmoe-stream --roster mini-a,mini-b --me mini-a --port 8301\nexpert-node --model-dir ~/models/olmoe-stream --roster mini-a,mini-b --me mini-b --port 8302\n\nexpert-net ~/models/olmoe-stream --nodes http://127.0.0.1:8301,http://127.0.0.1:8302 --chat\n```\n\nNodes bind `127.0.0.1`\n\nby default and have no authentication — pass\n`--host 0.0.0.0`\n\nonly on a trusted network. The decode loop is a LAN/metro\ndesign (per-layer round trips); the WAN's role is distributing the\ncontent-addressed expert partitions, never the token loop.\n\n```\npip install -e .\n\nmlx-sniper download qwen3-30b          # download + preprocess + calibrate\nmlx-sniper run  ~/models/qwen3-30b-stream -p \"Hello\" -v\nmlx-sniper chat ~/models/qwen3-30b-stream\nmlx-sniper serve ~/models/qwen3-30b-stream        # Ollama-compatible API\nmlx-sniper eval ~/models/qwen3-30b-stream          # perplexity on held-out text\n```\n\n`mlx-sniper download list`\n\nshows supported models (Qwen3-30B/Coder-30B,\nQwen3.5-35B/122B, Qwen3-Next-80B, Qwen3-235B; Gemma 4-26B experimental).\nAlready have an MLX 4-bit checkpoint? `mlx-sniper preprocess <src> <out>`\n\n.\n\n**Preprocess** splits the checkpoint into`pinned.safetensors`\n\n(RAM) and per-layer`bin/layer_XX.bin`\n\nfiles of 16 KB-aligned expert blocks (SSD).**Calibrate**(one-time) right-sizes the expert LRU cache for your RAM, records a cross-layer co-activation matrix, and sweeps the routing bias with a perplexity gate: a bias only ships if held-out perplexity stays within 5% of the unbiased baseline, measured on the same forward pass that serves.**Serve** runs the model with threaded`pread`\n\nprefetch of the next layer's predicted + selected experts, fused active-expert FFN via`gather_qmm`\n\n, and the router nudged toward already-cached experts.\n\nQwen3-30B-A3B 4-bit (17 GB on disk, 128 experts), M4 Mac Mini 16 GB with other apps running, 886-expert (2.3 GB) cache, 100-token generations, decode-mode perplexity on held-out text:\n\n| Routing bias | Speed | Decode ppl | Cache hit rate |\n|---|---|---|---|\n| 0.0 (default) | 1.15 tok/s | 4.18 | 51% |\n0.5 (opt-in: `--ppl-tolerance 1.10` ) |\n1.70 tok/s (+48%) | 4.42 (+5.7%) | 62% |\n| 1.0 | — | 6.38 (+53%) | quality cliff |\n| 1.5 | — | 14.11 (+238%) | quality cliff |\n\nTTFT ~11 s; sustained SSD streaming 1.7–2.3 GB/s at 1.2–1.9 ms/expert.\n\nOLMoE-1B-7B (3.6 GB, 64 experts — the demo model), same machine:\n**14.9 tok/s single-machine streaming** (85% hit rate, TTFT 1.3 s) and\n**19.3 tok/s as a two-node Expert Network**. Prefetch is early-router\nprediction (layer *i+1*'s pinned router run on layer *i*'s hidden state —\n97% recall@16 measured), with speculative reads parked in a victim buffer\nand promoted to the main cache only when a demand uses them.\n\nThree honest findings from re-measuring on the fixed code (details in RESEARCH.md):\n\n**The routing bias trades quality for speed**— it is not free. Earlier releases shipped bias 1.0–1.5 as a \"sweet spot\" validated by a two-prompt substring check; decode-mode perplexity shows 1.0+ degrades the model badly. v0.2 defaults to bias 0 and gates any bias on measured perplexity.**Prefill perplexity cannot see bias damage**(prefill activates ~80% of experts per layer, so the cache-aware bias barely engages) — the calibration gate must measure token-by-token decode.**Co-activation prefetch is bandwidth-neutral here**: fixing the bug that discarded its reads doubled consumed prefetches (1,810 → 3,407 per 100 tokens) at unchanged tok/s. The old \"70% prediction accuracy → speedup\" framing was not realizable.\n\nPractical note: on a 16 GB machine, cache sizing must respect *available*\nRAM, not total — an oversized expert cache pushes the OS into swap on the\nsame SSD the experts stream from, and throughput collapses ~50× (measured\n0.21 tok/s vs 1.7). `calibrate`\n\nhandles this automatically.\n\nEarlier published figures (5.37 tok/s 35B, 3.34 tok/s 30B, \"92% cache hit\") were measured on code with corrupted hit-rate accounting and a bias level that decode-ppl shows was damaging quality; treat them as superseded.\n\nAn `--expert-cache-size`\n\nmadvise patch for llama.cpp (see `llama-cpp/`\n\n)\nproduced 0.57 tok/s for a 30B MoE on an 8 GB M2 Air where stock llama.cpp\nproduced no output.\n\n-\n`src/mlx_expert_sniper/`\n\n— the pip-installable package (MLX, Apple Silicon) -\n`llama-cpp/`\n\n— expert-cache patch for llama.cpp (cross-platform, GGUF) — original sources + an`apply.sh`\n\nthat patches your own llama.cpp checkout -\n`sniper-router/`\n\n— thin client for driving a remote sniper/llama-server -\n`bench/`\n\n— research benchmark scripts behind the RESEARCH.md numbers -\n`tests/`\n\n— unit tests (`python -m pytest tests/`\n\n) -\n`RESEARCH.md`\n\n— full technical writeup -\n`src/expert_network/`\n\n— the Expert Network: nodes, binary protocol, distributed driver, HRW roster placement\n\nThe MLX engine is Apple Silicon, but Machine Yield is not. Any box running its own MoE engine — FreeToken on a DGX Spark or an RTX rig, vLLM, llama.cpp — joins through the sidecar, which proves what the yield system actually needs proven: that the machine holds the weights (content- addressed 4 MB chunk challenges), that it is alive (heartbeats + a health poll of the engine's OpenAI-compatible endpoint), and what it serves.\n\n```\n# on the Spark: FreeToken serving GLM/DeepSeek on :8000, then\nexpert-sidecar --model-path ~/models/GLM-4.6 --engine-url http://127.0.0.1:8000     --join <api-key> --advertise-url https://your-tunnel.example.com\n\n# once per model, publish its chunk manifest:\nexpert-sidecar --model-path ~/models/GLM-4.6 --write-manifest\n```\n\nStdlib-only — no mlx, no extra deps. Expert-partition serving on CUDA (the full network tier) is future work; see the issues.\n\nThe same sidecar makes the box a **Fast Token draft node**: it proxies\n`/v1/*`\n\nto its engine, so fast dense hardware proposes tokens and the\nexpert mesh verifies the batch in one forward. Any driver opts in:\n\n```\nexpert-net ~/models/... --nodes ... --spec --draft-url https://spark1.example.com/v1\n```\n\nMeasured over this path, twice:\n\n- localhost sidecar proxy (0.6B drafting 30B): 4.0 tok/forward, 41% acceptance\n**over the real internet**— a 6-vCPU VPS in a European datacenter (llama.cpp, 44 tok/s draft speed) drafting for a 30B on a Mac mini:** 5.0 tok/forward at 56% acceptance, 12/12 rounds remote**— no acceptance loss vs an in-process draft. The draft runs wherever hardware is fastest and cheapest; the verify runs where the experts live.\n\n`mlx-sniper serve`\n\nand the agent CLIs bind`127.0.0.1`\n\nby default. There is no authentication layer — do not expose them to untrusted networks.- The agent tools (\n`llama-cpp/sniper.py`\n\n,`sniper-router/router.py`\n\n) can execute model-proposed shell commands (`/shell`\n\n). Use with prompts and models you trust.\n\nMIT (see LICENSE). llama.cpp integration notes in NOTICE.", "url": "https://wpnews.pro/news/make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model", "canonical_source": "https://github.com/walter-grace/expert-sniper", "published_at": "2026-08-25 22:01:37+00:00", "updated_at": "2026-08-25 22:16:02.325172+00:00", "lang": "en", "topics": ["machine-learning", "artificial-intelligence", "ai-infrastructure", "ai-tools", "developer-tools"], "entities": ["Expert Sniper", "Qwen3-Coder-30B-A3B", "OLMoE-1B-7B", "M4 mini", "Apple", "MoE"], "alternates": {"html": "https://wpnews.pro/news/make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model", "markdown": "https://wpnews.pro/news/make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model.md", "text": "https://wpnews.pro/news/make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model.txt", "jsonld": "https://wpnews.pro/news/make-your-own-darkbloom-the-expert-network-pools-macs-into-one-model.jsonld"}}