DeepSeek-V4-Flash (284B MoE) at ~28 tok/s on 1x RTX 5090 + dual Xeon Gold 6138 — recipe + benchmarked negatives (companion to ktransformers#2088) A developer achieved ~28 tok/s single-stream decode of DeepSeek-V4-Flash (284B MoE) on a single RTX 5090 with dual Xeon Gold 6138 CPUs using a CPU-offloaded expert recipe. The setup uses ktransformers and sglang, with int4 experts computed on CPU and NUMA-sharded for locality, yielding a 14% speedup over the prior baseline while maintaining coding quality. A reproducible recipe + benchmark study — not a new model, not a novel technique . The core stack is stock kvcache-ai/ktransformers https://github.com/kvcache-ai/ktransformers + sglang. The value here is the measured results and the negatives : what to run, what to skip, and why, on an accessible single-GPU + dual-Xeon box. Companion to the upstream docs PR this study fed back: merged — AMXINT4 on Skylake no AMX , dual-Xeon tuning, and the single-stream negative results. ktransformers 2088 A serving recipe that runs DeepSeek-V4-Flash — 284B total / 13B active-parameter Mixture-of-Experts 256 routed experts/layer, top-6, 43 layers, MLA + sparse attention, built-in MTP head — for single-stream decode at ~28 tokens/sec, at 4-bit, with coding quality intact , on a single commodity workstation. It is OpenAI-API-compatible and drives agentic tool-calling. The model is 142 GB at 4-bit. It does not fit the GPU. This recipe runs it anyway experts computed on CPU . | GPU | 1× RTX 5090, 32 GB sm 120, 1.79 TB/s | | CPU | 2× Xeon Gold 6138 Skylake-SP, 40 physical cores, AVX-512 — no AMX tile, no VNNI | | RAM | 256 GB DDR4-2400, 8/12 channels ~100 GB/s realizable , GPU on NUMA node 0 | CPU-offloaded experts. Attention + a few hot experts run on the GPU; the 256 routed experts per layer are computed on the CPU ktransformers/kt-kernel , in RAM. Crucially the expert weights never cross PCIe — only small activations do. This is what lets a 142 GB model run on a 32 GB card. Unified stack. sglang scheduler, attention, MTP, OpenAI API + ktransformers kt-kernel CPU MoE , integrated via kt ep wrapper . One server, one endpoint. Low-bit. Experts stored/computed at int4 "AMXINT4" kernel — despite the name it needs only AVX512F+AVX512BW, so it runs on Skylake with no AMX tile . 4-bit is the near-lossless floor; coding quality holds. Locality-optimized. Experts are NUMA-sharded one shard per socket and read node-locally, so each socket reads from its own RAM instead of half-crossing the inter-socket link. This is where the 25 → ~28 tok/s +14% came from, at zero quality cost. The AMXINT4+NUMA recipe itself is documented upstream; what's added here is the quantified gain on this specific hardware. Plus: top-4 adaptive-k cull each token's experts from top-6 to top-4 — coding-verified , expert-deferral CPU/GPU overlap, and frequency-based hot-expert placement. | Config | tok/s | Quality | Notes | |---|---|---|---| | Original MXFP4 + GPU experts + EAGLE | 25 | ✓ | prior baseline | This recipe AMXINT4 numa, CPU-only, top-4 | ~28 | ✓ coding holds | +14%, simpler | Coding canary passes is palindrome, two sum, binary search, valid parens, merge intervals, fib, longest common prefix, group anagrams all correct; math/primes correct — see codegate.py . Decode is DDR4-bandwidth-saturated at this point — confirmed the ceiling. Re-verified 2026-07-22 on the same box: 27.9–28.0 tok/s steady-state decode streaming measurement, time between first and last token, prefill excluded — the same "gen throughput" the 28 refers to . | Lever | Result | Why it lost | |---|---|---| | EAGLE speculative decode | 27.2 | spec verifies a batch of draft tokens routing to different experts → multiplies the CPU expert reads that are the bottleneck | | More GPU experts | 27.5 | enabling machinery costs more than offloading 6/256 experts saves | | cpuinfer=80 hyperthreads | 2.7 | HT oversubscription thrashes the bandwidth-bound AVX-512 kernel; 40 = 1 thread/physical-core is optimal | | Sub-4-bit expert reduction