Make your own Darkbloom – The Expert Network pools Macs into one model

A new open-source project, Expert Sniper, pools ordinary Macs into a network that runs Mixture-of-Experts (MoE) models none of them could run alone, achieving 4.0 tok/s on a 16 GB M4 mini running Qwen3-Coder-30B-A3B 4-bit (17 GB on disk) and 19.3 tok/s on a two-node localhost setup with OLMoE-1B-7B. The project exploits MoE sparsity to stream only active experts from SSD or remote nodes, with each node owning a partition of experts via rendezvous hashing, and it is available via pip install with commands like mlx-sniper download and expert-node.

Connect ordinary computers into a network that runs MoE models none of them could run alone. MoE Mixture-of-Experts models activate only 8 of 128-256 experts per token, so ~97% of the weights are unused per computation. That sparsity has a consequence most inference stacks ignore: the experts don't have to live on the machine doing the thinking. They can live on any machine — on its SSD, served over the network — because what moves between boxes is activations ~4 KB per layer , while what moves within a box is expert weights ~13.5 MB per layer . Interconnect is never the bottleneck; pooled SSD bandwidth is the resource, and it scales linearly with every machine that joins: feasible when N x SSD bandwidth x cache hit rate bytes per token x target tok/s └ number of machines in the network Expert Sniper builds this in two tiers: One machine src/mlx expert sniper/ : pin the small always-needed weights attention, norms, router — ~0.9-1.4 GB in RAM and stream only the active experts from SSD via F NOCACHE + pread , with a right-sized LRU expert cache and threaded prefetch. A 17-21 GB model runs in a few GB of RAM. A network of machines src/expert network/ : each node owns a partition of the experts, loaded from its own SSD into its own RAM; a driver runs attention locally and dispatches expert compute to whichever nodes own the active experts. Every machine added is more expert RAM and more SSD bandwidth. Partition assignment is rendezvous hashing over a shared node roster — every node derives the same assignment with zero coordination, and adding a node moves only ~1/N of the experts. One machine, measured 16 GB M4 mini, Qwen3-Coder-30B-A3B 4-bit, 17 GB on disk : 4.0 tok/s, 72.5% expert-cache hit rate, 5.4 GB/s effective reads, ~3.6 GB peak RAM. The model is larger than the machine's memory. Two nodes + driver localhost, OLMoE-1B-7B, 64 experts split 32/32 by roster : 19.3 tok/s, TTFT 0.6 s, 2.9 ms/layer round trip, ~0.24 MB of network traffic per token — faster than the same model streaming from SSD on one machine 14.9 tok/s , because resident partitions take the SSD out of the token loop entirely. That measurement is localhost. Across two physical machines it is a well-founded projection, not a result; see issue 3. Joining does not mean downloading the whole model. A node fetches only the experts it owns, from machines already serving them — the /block/{layer}/{expert} endpoint that proves a node holds an expert is the same one that hands it to the next node, and every block is checked against a published sha256 manifest on arrival. Blocks land in sparse files at the format's fixed offsets, so the engine reads them unchanged while the disk holds only this machine's slice measured: 0.2 GB on disk for a partition of a 4 GB model . pip install -e ". network " mlx-sniper download olmoe-1b-7b -o ~/models/olmoe-stream 3.6 GB demo model each machine or terminal derives its partition from the shared roster: expert-node --model-dir ~/models/olmoe-stream --roster mini-a,mini-b --me mini-a --port 8301 expert-node --model-dir ~/models/olmoe-stream --roster mini-a,mini-b --me mini-b --port 8302 expert-net ~/models/olmoe-stream --nodes http://127.0.0.1:8301,http://127.0.0.1:8302 --chat Nodes bind 127.0.0.1 by default and have no authentication — pass --host 0.0.0.0 only on a trusted network. The decode loop is a LAN/metro design per-layer round trips ; the WAN's role is distributing the content-addressed expert partitions, never the token loop. pip install -e . mlx-sniper download qwen3-30b download + preprocess + calibrate mlx-sniper run ~/models/qwen3-30b-stream -p "Hello" -v mlx-sniper chat ~/models/qwen3-30b-stream mlx-sniper serve ~/models/qwen3-30b-stream Ollama-compatible API mlx-sniper eval ~/models/qwen3-30b-stream perplexity on held-out text mlx-sniper download list shows supported models Qwen3-30B/Coder-30B, Qwen3.5-35B/122B, Qwen3-Next-80B, Qwen3-235B; Gemma 4-26B experimental . Already have an MLX 4-bit checkpoint? mlx-sniper preprocess <src <out . Preprocess splits the checkpoint into pinned.safetensors RAM and per-layer bin/layer XX.bin files of 16 KB-aligned expert blocks SSD . Calibrate one-time right-sizes the expert LRU cache for your RAM, records a cross-layer co-activation matrix, and sweeps the routing bias with a perplexity gate: a bias only ships if held-out perplexity stays within 5% of the unbiased baseline, measured on the same forward pass that serves. Serve runs the model with threaded pread prefetch of the next layer's predicted + selected experts, fused active-expert FFN via gather qmm , and the router nudged toward already-cached experts. Qwen3-30B-A3B 4-bit 17 GB on disk, 128 experts , M4 Mac Mini 16 GB with other apps running, 886-expert 2.3 GB cache, 100-token generations, decode-mode perplexity on held-out text: | Routing bias | Speed | Decode ppl | Cache hit rate | |---|---|---|---| | 0.0 default | 1.15 tok/s | 4.18 | 51% | 0.5 opt-in: --ppl-tolerance 1.10 | 1.70 tok/s +48% | 4.42 +5.7% | 62% | | 1.0 | — | 6.38 +53% | quality cliff | | 1.5 | — | 14.11 +238% | quality cliff | TTFT ~11 s; sustained SSD streaming 1.7–2.3 GB/s at 1.2–1.9 ms/expert. OLMoE-1B-7B 3.6 GB, 64 experts — the demo model , same machine: 14.9 tok/s single-machine streaming 85% hit rate, TTFT 1.3 s and 19.3 tok/s as a two-node Expert Network . Prefetch is early-router prediction layer i+1 's pinned router run on layer i 's hidden state — 97% recall@16 measured , with speculative reads parked in a victim buffer and promoted to the main cache only when a demand uses them. Three honest findings from re-measuring on the fixed code details in RESEARCH.md : The routing bias trades quality for speed — it is not free. Earlier releases shipped bias 1.0–1.5 as a "sweet spot" validated by a two-prompt substring check; decode-mode perplexity shows 1.0+ degrades the model badly. v0.2 defaults to bias 0 and gates any bias on measured perplexity. Prefill perplexity cannot see bias damage prefill activates ~80% of experts per layer, so the cache-aware bias barely engages — the calibration gate must measure token-by-token decode. Co-activation prefetch is bandwidth-neutral here : fixing the bug that discarded its reads doubled consumed prefetches 1,810 → 3,407 per 100 tokens at unchanged tok/s. The old "70% prediction accuracy → speedup" framing was not realizable. Practical note: on a 16 GB machine, cache sizing must respect available RAM, not total — an oversized expert cache pushes the OS into swap on the same SSD the experts stream from, and throughput collapses ~50× measured 0.21 tok/s vs 1.7 . calibrate handles this automatically. Earlier published figures 5.37 tok/s 35B, 3.34 tok/s 30B, "92% cache hit" were measured on code with corrupted hit-rate accounting and a bias level that decode-ppl shows was damaging quality; treat them as superseded. An --expert-cache-size madvise patch for llama.cpp see llama-cpp/ produced 0.57 tok/s for a 30B MoE on an 8 GB M2 Air where stock llama.cpp produced no output. - src/mlx expert sniper/ — the pip-installable package MLX, Apple Silicon - llama-cpp/ — expert-cache patch for llama.cpp cross-platform, GGUF — original sources + an apply.sh that patches your own llama.cpp checkout - sniper-router/ — thin client for driving a remote sniper/llama-server - bench/ — research benchmark scripts behind the RESEARCH.md numbers - tests/ — unit tests python -m pytest tests/ - RESEARCH.md — full technical writeup - src/expert network/ — the Expert Network: nodes, binary protocol, distributed driver, HRW roster placement The MLX engine is Apple Silicon, but Machine Yield is not. Any box running its own MoE engine — FreeToken on a DGX Spark or an RTX rig, vLLM, llama.cpp — joins through the sidecar, which proves what the yield system actually needs proven: that the machine holds the weights content- addressed 4 MB chunk challenges , that it is alive heartbeats + a health poll of the engine's OpenAI-compatible endpoint , and what it serves. on the Spark: FreeToken serving GLM/DeepSeek on :8000, then expert-sidecar --model-path ~/models/GLM-4.6 --engine-url http://127.0.0.1:8000 --join <api-key --advertise-url https://your-tunnel.example.com once per model, publish its chunk manifest: expert-sidecar --model-path ~/models/GLM-4.6 --write-manifest Stdlib-only — no mlx, no extra deps. Expert-partition serving on CUDA the full network tier is future work; see the issues. The same sidecar makes the box a Fast Token draft node : it proxies /v1/ to its engine, so fast dense hardware proposes tokens and the expert mesh verifies the batch in one forward. Any driver opts in: expert-net ~/models/... --nodes ... --spec --draft-url https://spark1.example.com/v1 Measured over this path, twice: - localhost sidecar proxy 0.6B drafting 30B : 4.0 tok/forward, 41% acceptance over the real internet — a 6-vCPU VPS in a European datacenter llama.cpp, 44 tok/s draft speed drafting for a 30B on a Mac mini: 5.0 tok/forward at 56% acceptance, 12/12 rounds remote — no acceptance loss vs an in-process draft. The draft runs wherever hardware is fastest and cheapest; the verify runs where the experts live. mlx-sniper serve and the agent CLIs bind 127.0.0.1 by default. There is no authentication layer — do not expose them to untrusted networks.- The agent tools llama-cpp/sniper.py , sniper-router/router.py can execute model-proposed shell commands /shell . Use with prompts and models you trust. MIT see LICENSE . llama.cpp integration notes in NOTICE.