FreeToken Unlocks Frontier MoE Inference on Consumer Hardware via Dynamic Co-Execution

Researchers from UC Berkeley and MIT released FreeToken, an open-source inference engine that enables frontier Mixture-of-Experts (MoE) models to run on consumer hardware via dynamic co-execution, achieving 3–4x faster decode and 6–30x faster prefill compared to Ollama and llama.cpp. The system, co-authored by Databricks co-founders Matei Zaharia and Ion Stoica, uses a q* policy to split token computation between CPU and GPU in real time, and benchmarks show it ran Qwen3.6-35B at ~39 tokens/sec on an 8GB RTX 4060 laptop and served DeepSeek-V4-Flash (284B) on an RTX 5090 desktop.

Researchers from UC Berkeley and MIT have introduced FreeToken https://github.com/FlashML-org/FreeToken , an open-source inference engine designed to bridge the gap between frontier Mixture-of-Experts MoE models and consumer-grade hardware. Co-authored by Databricks co-founders Matei Zaharia https://www.linkedin.com/in/mateizaharia and Ion Stoica https://www.linkedin.com/in/ionstoica alongside Song Han https://hanlab.mit.edu/songhan , Kurt Keutzer https://www.linkedin.com/in/kurtkeutzer/ and others, the project shifts the paradigm of edge AI from treating personal machines as constrained datacenter nodes to managing them as elastic, heterogeneous computing fabrics. While sparse MoE architectures compute only a fraction of total parameters per token, decoding requires routing across hundreds of billions of inactive weights. In datacenter environments, high-bandwidth interconnects like NVLink mask expert transfer overhead. On consumer hardware, however, PCIe throughput typically 16–64 GB/s and host RAM latency form severe decode bottlenecks. Existing edge runtimes rely on static expert offloading where inactive weights reside in system RAM and are synchronously streamed to the GPU upon activation, completely stalling execution on cache misses. To solve this, FreeToken replaces rigid offloading with a dynamic co-scheduling formulation termed the q policy https://arxiv.org/abs/2608.16157 . Rather than halting the GPU during cache misses, FreeToken splits token computation between CPU cores and GPU tensor cores according to real-time interconnect throughput. The system employs a fast weight format FTW alongside full-layer double buffering, allowing weight streaming over PCIe to overlap entirely with active computation layers. An elastic memory manager also dynamically reallocates VRAM between KV cache entries and resident expert slots during runtime without triggering model reloads. Modern coding assistants and autonomous agents introduce unique execution patterns: frequent prompt modifications, tool-call responses, and thinking blocks constantly alter the context window. Standard engines discard linear KV caches when prefixes mutate, triggering costly full-sequence recomputations. FreeToken integrates semantic anchor checkpointing, caching intermediate attention states and recurrent activations at logical task boundaries. When an agent edits intermediate tool arguments or injects external execution output, FreeToken reuses existing sub-sequence states instead of invalidating the prompt cache. FreeToken Overview, Source: Figure 1 "FreeToken: EfficientEdge-NativeMoEServingwith Bandwidth-AdaptiveExecution" Research paper This architecture distinctly separates FreeToken from other runtimes across the ecosystem: Ollama https://github.com/ollama/ollama and llama.cpp https://github.com/ggml-org/llama.cpp : Optimised for GGUF quantisation and layer-wise offloading, but lack dynamic load splitting for sparse experts across host and device. FreeToken achieves 3–4x faster decode and 6–30x faster prefill on equivalent MoE models. vLLM https://github.com/vllm-project/vllm and SGLang https://github.com/sgl-project/sglang : Tailored for datacenter throughput via PagedAttention and continuous batching, assuming high interconnect bandwidth rather than heterogeneous memory hierarchies. KTransformers https://github.com/kvcache-ai/ktransformers : Employs static CPU/GPU offloading rules, whereas FreeToken computes closed-form optimal splits per layer in real time. According to the benchmarks section of the paper, FreeToken ran Qwen3.6-35B at ~39 tokens/sec on an 8GB RTX 4060 laptop, served DeepSeek-V4-Flash 284B on an RTX 5090 desktop, and processed GLM-5.2 753B on a single workstation GPU. The CLI and desktop client are available via FlashML.ai https://www.flashml.ai/ and the GitHub repository https://github.com/FlashML-org/FreeToken , supporting NVIDIA RTX 30, 40, and 50 series GPUs on Linux and Windows. Community sentiment across Hacker News https://news.ycombinator.com/item?id=49434580 and Reddit's LocalLLaMA forum https://www.reddit.com/r/LocalLLaMA/comments/1vv6v00/freetokens project is impressive/ highlights growing excitement around local hardware sovereignty, tempered by technical scrutiny over real-world edge scheduling. On Hacker News, engineers point out that combining bandwidth-adaptive MoE serving with affordable consumer memory such as used RTX 3090/4080 GPUs paired with standard DDR4/DDR5 RAM significantly lowers the barrier for self-hosting frontier-class reasoning agents without recurring cloud API fees. Meanwhile, the LocalLLaMA benchmark analysis https://www.reddit.com/r/LocalLLaMA/comments/1vv6v00/freetokens project is impressive/ and paper discussion thread https://www.reddit.com/r/LocalLLaMA/comments/1vxjcsw/260816157 freetoken efficient edgenative moe/ have spurred deep technical debates over whether theoretical q closed-form calculations accurately reflect real-world CPU dispatch latency, memory contention, and varying expert residency under concurrent agent workloads. Despite debates over baseline comparisons against hand-tuned llama.cpp setups, the broader consensus signals a definitive paradigm shift: developers increasingly view heterogeneous edge orchestration as essential for escaping proprietary API lock-in, cutting agent iteration costs to zero, and preserving IP privacy in automated coding workflows.