FreeToken reports 22-25 tok/s for a 284B DeepSeek model on one RTX 5090

FreeToken, an open-source inference engine developed by UC Berkeley researchers including Shuo Yang and Xiaoze Fan, reports decoding DeepSeek-V4-Flash, a 284B-parameter Mixture-of-Experts model, at 22-25 tokens per second on a single RTX 5090 desktop with 32 GB VRAM, 192 GiB system memory, and an AMD Ryzen 9 9950X3D CPU. The paper also reports Qwen3.6-35B-A3B at 39.3 tokens per second on an RTX 4060 laptop GPU and GLM-5.2 at 14.9 tokens per second on an RTX PRO 6000. The results suggest MoE scheduling can enable large-model inference on high-end PCs, though benchmarks have not been independently replicated.

FreeToken reports 22-25 tok/s for a 284B DeepSeek model on one RTX 5090 Shuo Yang, Xiaoze Fan and collaborators report the decode rate using CPU compute, a 32 GB RTX 5090 and 192 GiB of system memory. By RuntimeWire Staff /author/runtimewire-staff · Published Primary source: Aligned News - AI Intelligence https://x.com/cgtwts/status/2091149529274085436 Why it matters FreeToken's reported results suggest that MoE scheduling can move some large-model inference from GPU clusters onto high-end PCs, though the system still relies on substantial CPU and system-memory resources and its benchmarks have not been independently replicated. Shuo Yang https://andy-yang-1.github.io/?ref=runtimewire and Xiaoze Fan https://jasonfxz.top/?ref=runtimewire are the equal-contribution first authors of a new paper on FreeToken, an open-source inference engine that pools CPU, system-memory and GPU resources to run Mixture-of-Experts models on local hardware. The paper reports 22 to 25 decode tokens per second for DeepSeek-V4-Flash /models/azure/deepseek-v4-flash on an RTX 5090 desktop, while the system uses CPU compute and 192 GiB of system memory alongside the GPU. The result appears in the team's research paper https://arxiv.org/pdf/2608.16157?ref=runtimewire , produced by researchers affiliated with UC Berkeley and other institutions. The preprint was submitted on August 17, 2026 https://arxiv.org/abs/2608.16157?ref=runtimewire . The same paper reports Qwen3.6-35B-A3B /models/qwen/qwen3.6-35b-a3b at 39.3 tokens per second on an RTX 4060 laptop GPU with 8 GB of VRAM, and GLM-5.2 /models/z-ai/glm-5.2:batch at 14.9 tokens per second on one 96 GB RTX PRO 6000. Yang is a UC Berkeley EECS Ph.D. student advised by Ion Stoica https://www2.eecs.berkeley.edu/Faculty/Homepages/stoica.html?ref=runtimewire whose work covers language-model serving, GPU kernels and multimodal systems. Fan is a Shanghai Jiao Tong University undergraduate and a visiting student researcher at Berkeley's Sky Computing Lab. The project also includes Melissa Pan https://melissa-pan.github.io/?ref=runtimewire , Haocheng Xi https://haochengxi.github.io/?ref=runtimewire , Zhe Wang and Shanlin Sun, alongside senior researchers Kurt Keutzer https://people.eecs.berkeley.edu/~keutzer/?ref=runtimewire , Song Han https://hanlab.mit.edu/songhan/?ref=runtimewire , Matei Zaharia https://people.csail.mit.edu/matei/?ref=runtimewire , Chenfeng Xu https://www.chenfengx.com/?ref=runtimewire and Stoica. One GPU, plus the rest of the PC The headline number needs its denominator. Table 1 of the paper lists the RTX 5090 desktop as using an AMD Ryzen 9 9950X3D processor, 192 GiB of DDR5 system memory and a GPU with 32 GB of VRAM https://arxiv.org/pdf/2608.16157?ref=runtimewire . The project treats GPU VRAM, CPU compute, system memory, PCIe bandwidth and local storage as a combined serving platform instead of requiring the full model to fit in GPU memory. DeepSeek-V4-Flash https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash?ref=runtimewire is a sparse Mixture-of-Experts model from DeepSeek-AI with 284 billion total parameters and roughly 13 billion active parameters per token. The FreeToken paper says each token is routed through six of 256 experts in each of 43 layers https://arxiv.org/pdf/2608.16157?ref=runtimewire . It also says the active computation can fit within the RTX 5090's 32 GB of VRAM, while the complete expert pool remains much larger https://arxiv.org/pdf/2608.16157?ref=runtimewire . FreeToken's paper describes CPU-GPU co-execution, expert caching and bandwidth-adaptive scheduling https://arxiv.org/pdf/2608.16157?ref=runtimewire . The authors say the scheduler assigns work according to available CPU and GPU bandwidth when an expert is absent from the GPU cache. The system consequently depends on the surrounding PC, including its memory capacity, memory bandwidth and PCIe connection. The laptop result carries the same qualification. The paper lists the RTX 4060 machine as using an Intel Core i9-13900H processor and 32 GiB of LPDDR5 memory alongside the GPU's 8 GB of VRAM https://arxiv.org/pdf/2608.16157?ref=runtimewire . It reports 39.3 tokens per second for an NVFP4 version of Qwen3.6-35B-A3B under a coding workload https://arxiv.org/pdf/2608.16157?ref=runtimewire . The paper labels the metric "decode throughput per-request mean tok/s ." That measurement does not specify total elapsed time from prompt submission through completion. FreeToken schedules the whole PC FreeToken's contribution is a set of scheduling and caching choices built around the uneven resources in consumer hardware. During prompt processing, the paper says the engine uses full-layer double buffering https://arxiv.org/pdf/2608.16157?ref=runtimewire , transferring data for an upcoming layer while the GPU works on the current one. For token generation, it describes a bandwidth-adaptive policy that assigns cache misses between PCIe transfers and CPU execution https://arxiv.org/pdf/2608.16157?ref=runtimewire . A global least-recently-used cache keeps recently selected experts in VRAM. The paper describes semantic-aware recurrent-state checkpointing for agent sessions https://arxiv.org/pdf/2608.16157?ref=runtimewire . The engine places checkpoints at thinking segments, tool calls and outputs, and conversation turns, allowing it to re-prefill only a new suffix after an edit. The authors cite OpenClaw, OpenCode and SWE-agent as workloads where that state reuse can matter. The authors also say FreeToken can rebuild its GPU expert cache without restarting the engine or reloading the host-resident expert pool https://arxiv.org/pdf/2608.16157?ref=runtimewire . These features reflect the project's intended workload: long-running local agents, where repeated prompts, edits and cache behavior affect performance. The benchmarks favor FreeToken, under FreeToken's test conditions Sections 5.1 and 5.2 of the paper evaluate six GPU systems across four workloads, with comparisons against llama.cpp, Ollama, KTransformers and MoE-Infinity https://arxiv.org/pdf/2608.16157?ref=runtimewire . The workloads cover AIME math reasoning, OpenCode and Claude Code working on a SWE-bench issue, and an OpenClaw email-and-calendar task. The paper reports 77 to 83 tokens per second for Qwen3.6-35B-A3B and 22 to 25 tokens per second for DeepSeek-V4-Flash on the RTX 5090 systems https://arxiv.org/pdf/2608.16157?ref=runtimewire . Those figures are the authors' reported measurements from a newly posted preprint, not an independently replicated production-latency result. Across five consumer systems, the paper reports per-request mean decode-throughput gains of 1.3x to 2.1x over the strongest baseline in each test https://arxiv.org/pdf/2608.16157?ref=runtimewire . The machine-level figures are 1.3x on the RTX 3090 and RTX 4090, 1.9x on a rented RTX 5090 server, 2.1x on the RTX 5090 desktop and 1.8x on the RTX 4060 laptop. The paper separately reports a 1.5x to 2.3x range across workloads on its RTX 5090 systems. The comparisons do not establish a universal speed ranking. The paper says Ollama and MoE-Infinity lacked DeepSeek-V4 support, while MoE-Infinity served only the first workload and did not provide a usable multi-turn agent server https://arxiv.org/pdf/2608.16157?ref=runtimewire . Baseline coverage therefore varied by model and workload. Agent trajectories could also diverge, complicating wall-clock comparisons across systems. The reported results should be read as decode-throughput measurements under the authors' test conditions. Sustained performance will depend on model format and quantization, prompt and output lengths, RAM and PCIe bandwidth, CPU capacity, cache behavior and other work running on the machine. Open weights still need an inference system The team published the FreeToken source code https://github.com/FlashML-org/FreeToken?ref=runtimewire under the Apache 2.0 license. The repository displayed about 1,000 GitHub stars as of August 22, 2026. It says the engine supports more than 20 MoE models, Windows and Linux, a command-line interface, and APIs compatible with OpenAI and Anthropic clients. The repository also directs users to FlashML for a desktop app, while the public software remains labeled beta https://www.flashml.ai/?ref=runtimewire . FreeToken addresses a specific gap in open AI. Downloadable weights provide access to a model, while serving those weights at useful speeds can still require datacenter GPUs or paid APIs. The research team is testing whether sparse models and coordinated scheduling across CPU, GPU and memory can move some of that work onto developer workstations. The reported 284-billion-parameter deployment spans the RTX 5090, a high-end CPU and a 192 GiB system-memory pool. The complete model does not reside in the GPU's 32 GB of VRAM. That makes the result relevant to local-inference developers, though it remains a demonstration on a heavily provisioned desktop and does not make the rest of the computer optional.