# KTransformers Turned CPU Offloading Into Real Infrastructure > Source: > Published: 2026-07-24 09:08:43+00:00 [AI](https://sourcefeed.dev/c/ai)Article # KTransformers Turned CPU Offloading Into Real Infrastructure The Tsinghua-born framework dissolved into SGLang and LLaMA-Factory, making trillion-class MoE models a single-machine problem. [Priya Nair](https://sourcefeed.dev/u/priya_nair) When [KTransformers](https://github.com/kvcache-ai/ktransformers) first blew up in early 2025, it was a party trick: run DeepSeek's 671B-parameter model on a desktop with one 24GB GPU by stashing most of the weights in system RAM. Impressive, barely usable, and easy to dismiss as a demo for people who refuse to pay for APIs. That dismissal no longer holds. The project — built by Tsinghua's MADSys lab with Approaching.AI, Apache-2.0 licensed, now around 19k stars — has quietly restructured itself from a standalone inference framework into something more durable: a set of CPU kernels and placement strategies that live *inside* the stacks you already use. And that pivot is the actual story, because it turns CPU offloading from a hobbyist workaround into a legitimate deployment strategy for the biggest open-weight models. ## Why MoE broke the old math The reason this works at all is that frontier open-weight models stopped being dense. DeepSeek-V3/R1, Kimi K2.5, GLM, MiniMax-M3, Qwen3-Next — every serious open release in the past year is a sparse mixture-of-experts. A 671B MoE might activate under 40B parameters per token. The rest of the weights are cold, sitting there occupying memory. Cold weights don't need HBM. They need capacity, and DRAM has capacity at roughly a tenth of the price. KTransformers' bet is a clean split: attention and shared experts — dense, compute-hungry — stay on the GPU; the sparsely-activated routed experts go to CPU, with the hottest experts pinned back onto spare VRAM. On Intel Xeons the CPU side runs hand-tuned AMX kernels that the team claims sustain 21.3 TFLOPS — about 3.9× what PyTorch gets from the same silicon — switching to AVX-512 paths for decode, where the workload is bandwidth-bound rather than compute-bound. Those numbers come from the project and its partners, so apply the usual discount. But the architecture argument doesn't depend on the exact TFLOPS: sparse activation means most of a trillion-parameter model is idle at any moment, and paying HBM prices to store idle weights is the single biggest waste in self-hosted LLM serving. ## The framework that dissolved itself The smartest thing KTransformers did in the past year was stop competing with serving frameworks. As of the 0.6.x restructuring this spring, the original monolithic framework is archived and the project ships as two components: **kt-kernel**, the CPU expert-inference library that [SGLang](https://github.com/sgl-project/sglang) calls into, and **kt-sft**, a fine-tuning path that plugs into [LLaMA-Factory](https://github.com/hiyouga/LLaMA-Factory). The SGLang integration, announced on the [LMSYS blog](https://www.lmsys.org/blog/2025-10-22-KTransformers/) last October, is the production-facing half. You get SGLang's scheduler, radix cache, and OpenAI-compatible API, with expert offload as launch flags rather than a separate runtime: ``` python -m sglang.launch_server \ --model /path/to/gpu-weights \ --kt-amx-weight-path /path/to/cpu-weights \ --kt-cpuinfer 80 \ --kt-num-gpu-experts 200 \ --kt-amx-method AMXINT4 ``` On the benchmark LMSYS published — DeepSeek-R1 on 8× L20 GPUs, hardly a lavish setup — the hybrid path hit 227 tokens/s aggregate with ~300ms inter-token latency under concurrency. Not H100-cluster numbers, but these are inference-grade cards plus commodity Xeon RAM serving a 671B model. The fine-tuning half is arguably more disruptive. Through the LLaMA-Factory integration, kt-sft runs LoRA-style SFT on the full DeepSeek-671B with 2–4 RTX 4090s and a fat-RAM host — the team reports ~3.7 iterations/s on four 4090s, claiming 6–12× over ZeRO-Offload-class alternatives. Until now, "fine-tune a frontier-scale MoE" meant renting a multi-node H100 cluster or not doing it. If those numbers hold up in the wild, a research group with a €15k workstation is in the game. The cadence matters too: day-0 or near-day-0 support landed for DeepSeek-V4-Flash in May and MiniMax-M3 and GLM-5.2 in June. That's release-week responsiveness that llama.cpp's GGUF pipeline often takes weeks to match for exotic MoE architectures. ## What it actually takes, and who it's for Be clear-eyed about the hardware. The validated DeepSeek-V4-Flash recipe is one RTX 5090, an AVX-512 CPU, **256GB+ of system RAM**, and ~340GB of disk for MXFP4 weights, yielding 20+ tokens/s decode single-stream. That's "consumer GPU" only in the narrowest sense — the surrounding box is a workstation or a used dual-Xeon server, and the best kernels want AMX, which means Sapphire Rapids or newer. An AVX2 fallback exists, plus ROCm and Intel Arc backends, but the headline performance is an Intel-server story. Against llama.cpp — the incumbent for this crowd — the difference is architectural. llama.cpp offloads whole layers and treats the CPU as a slow GPU; KTransformers places *experts* individually, keeps attention entirely on GPU, and brings quantized-GEMM kernels built for server CPUs. For dense 70B models, llama.cpp remains the simpler answer. For trillion-class MoEs, expert-level placement is just the correct abstraction, and it shows in the throughput gap. The honest trade-offs: decode speed is capped by DRAM bandwidth, so concurrency scaling flattens far earlier than pure-GPU serving; 20 tok/s is fine for chat and painful for deep agentic loops that burn tens of thousands of tokens per task; and the setup — pre-quantized CPU weight formats, NUMA tuning, a forked-SGLang install for the newest models — is still research-lab ergonomics, not `docker run` . ## Verdict This is a genuine shift, but for a specific customer: teams that need frontier-scale open models on-premises — for privacy, data residency, or fine-tuning ownership — at single-machine cost. For high-QPS API serving, GPU fleets still win on tokens per dollar and it's not close. But the "you need eight H100s to even touch a 671B model" assumption is now simply false, for inference *and* for SFT, and KTransformers is the main reason. The project's decision to dissolve into SGLang and LLaMA-Factory rather than build a parallel universe is what makes me take it seriously: infrastructure that wants to be adopted, not admired. If you're running a Xeon box with RAM to spare, this went from curiosity to the default thing to try. ## Sources & further reading - [kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers)— github.com - [Accelerating Hybrid Inference in SGLang with KTransformers CPU Kernels](https://www.lmsys.org/blog/2025-10-22-KTransformers/)— lmsys.org - [KTransformers Fine-Tuning x LLaMA-Factory Integration](https://blog.llamafactory.net/en/posts/ktransformers/)— blog.llamafactory.net - [DeepSeek-V4-Flash Deployment Guide](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepSeek-V4-Flash.md)— github.com [Priya Nair](https://sourcefeed.dev/u/priya_nair)· AI & Developer Experience Writer Priya covers AI frameworks, developer productivity tooling, and the startup ecosystem across South and Southeast Asia, bringing a researcher's rigour and a practitioner's empathy to every story. She is deeply sceptical of benchmarks and asks hard questions so her readers don't have to. ## Discussion 0 No comments yet Be the first to weigh in.