{"slug": "deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe", "title": "DeepSeek-V4-Flash (284B MoE) at ~28 tok/s on 1x RTX 5090 + dual Xeon Gold 6138 — recipe + benchmarked negatives (companion to ktransformers#2088)", "summary": "A developer achieved ~28 tok/s single-stream decode of DeepSeek-V4-Flash (284B MoE) on a single RTX 5090 with dual Xeon Gold 6138 CPUs using a CPU-offloaded expert recipe. The setup uses ktransformers and sglang, with int4 experts computed on CPU and NUMA-sharded for locality, yielding a 14% speedup over the prior baseline while maintaining coding quality.", "body_md": "A reproducible recipe + benchmark study — **not a new model, not a novel technique**. The core stack is stock\n[kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers) + sglang. The value here is the measured\nresults and the **negatives**: what to run, what to skip, and why, on an accessible single-GPU + dual-Xeon box.\n\nCompanion to the upstream docs PR this study fed back:\n\n(merged) — AMXINT4 on Skylake (no AMX), dual-Xeon tuning, and the single-stream negative results.[ktransformers#2088]\n\nA serving recipe that runs **DeepSeek-V4-Flash** — 284B total / 13B active-parameter Mixture-of-Experts\n(256 routed experts/layer, top-6, 43 layers, MLA + sparse attention, built-in MTP head) — for **single-stream\ndecode at ~28 tokens/sec, at 4-bit, with coding quality intact**, on a single commodity workstation. It is\nOpenAI-API-compatible and drives agentic tool-calling.\n\nThe model is 142 GB at 4-bit. It does **not** fit the GPU. This recipe runs it anyway (experts computed on CPU).\n\n| GPU | 1× RTX 5090, 32 GB (sm_120, 1.79 TB/s) |\n| CPU | 2× Xeon Gold 6138 (Skylake-SP, 40 physical cores, AVX-512 — no AMX tile, no VNNI) |\n| RAM | 256 GB DDR4-2400, 8/12 channels (~100 GB/s realizable), GPU on NUMA node 0 |\n\n**CPU-offloaded experts.** Attention + a few hot experts run on the GPU; the 256 routed experts per layer are**computed on the CPU**(ktransformers/kt-kernel), in RAM. Crucially the expert*weights never cross PCIe*— only small activations do. This is what lets a 142 GB model run on a 32 GB card.**Unified stack.** sglang (scheduler, attention, MTP, OpenAI API) + ktransformers kt-kernel (CPU MoE), integrated via`kt_ep_wrapper`\n\n. One server, one endpoint.**Low-bit.** Experts stored/computed at**int4**(\"AMXINT4\" kernel — despite the name it needs only AVX512F+AVX512BW, so it runs on Skylake with no AMX tile). 4-bit is the near-lossless floor; coding quality holds.**Locality-optimized.** Experts are**NUMA-sharded**(one shard per socket) and read node-locally, so each socket reads from its own RAM instead of half-crossing the inter-socket link. This is where the 25 → ~28 tok/s (+14%) came from, at zero quality cost. (The AMXINT4+NUMA recipe itself is documented upstream; what's added here is the quantified gain on this specific hardware.)\n\nPlus: **top-4 adaptive-k** (cull each token's experts from top-6 to top-4 — coding-verified), expert-deferral\nCPU/GPU overlap, and frequency-based hot-expert placement.\n\n| Config | tok/s | Quality | Notes |\n|---|---|---|---|\n| Original (MXFP4 + GPU experts + EAGLE) | 25 | ✓ | prior baseline |\nThis recipe (AMXINT4 numa, CPU-only, top-4) |\n~28 |\n✓ coding holds |\n+14%, simpler |\n\nCoding canary passes (is_palindrome, two_sum, binary_search, valid_parens, merge_intervals, fib,\nlongest_common_prefix, group_anagrams all correct; math/primes correct — see `codegate.py`\n\n). Decode is\n**DDR4-bandwidth-saturated** at this point — confirmed the ceiling.\n\n*Re-verified 2026-07-22 on the same box: 27.9–28.0 tok/s steady-state decode (streaming measurement, time between\nfirst and last token, prefill excluded — the same \"gen throughput\" the 28 refers to).*\n\n| Lever | Result | Why it lost |\n|---|---|---|\n| EAGLE speculative decode | 27.2 | spec verifies a batch of draft tokens routing to different experts → multiplies the CPU expert reads that are the bottleneck |\n| More GPU experts | 27.5 | enabling machinery costs more than offloading 6/256 experts saves |\n| cpuinfer=80 (hyperthreads) | 2.7 | HT oversubscription thrashes the bandwidth-bound AVX-512 kernel; 40 = 1 thread/physical-core is optimal |\n| Sub-4-bit expert reduction (<top-4) | breaks coding | expert-count floor is top-4 |\n\n**Bottom line:** single-stream decode is DDR4-bandwidth-saturated at ~28 tok/s (4-bit). Going faster needs more\nmemory bandwidth (populate the empty DIMM channels) or sub-4-bit experts (which need a new kernel + quality care).\nPost-training 2-bit/ternary breaks coding on DSV4 experts (~4× the weight error of int4); good ternary needs QAT.\n\nThe recipe is stock ktransformers + sglang — but **building that stack for the RTX 5090 (sm_120) is the actual work.**\nPin to the exact versions this was validated on; upstream flag names (`--kt-method`\n\n, `SGLANG_DSV4_MODE`\n\n) can drift on\n`main`\n\n.\n\n**Hard minimums (not optional — check these before downloading 150 GB of weights):**\n\n| Resource | Minimum | Why |\n|---|---|---|\n| System RAM | ~250 GB (256 GB here) |\nExperts are CPU-resident: 138 GB converted weights + ~165 GB serving RSS. A 128 GB box cannot load it. |\n| Free disk | ~300 GB | ~150 GB source download + 138 GB converted shards |\n| CPU | x86-64 with AVX512F + AVX512BW |\nThe \"AMXINT4\" int4 kernel needs these — but NOT the AMX tile or VNNI (verified on Skylake-SP that has neither). |\n| GPU | CUDA GPU, ~24 GB VRAM used at 8K ctx | Holds attention + KV only; experts stay on CPU. sm_120 for the 5090. |\n\nDual-socket NUMA is what buys the +14% locality win, but a single socket works (the recipe just collapses to one shard).\n\n**Validated environment (pin these):**\n\n**ktransformers @**—`7c021b430c36a408032c20bbf3833dc1bce6efa4`\n\n*verified stock: this recipe runs with*(confirmed clean working tree; the top-4 override and AMXINT4 path are all upstream).**zero** local patches- sglang submodule @\n`1e098a77ba395dc1a5f2dcbdf57bdb188e84bcee`\n\n**torch**`2.9.1+cu128`\n\n, CUDA`12.8`\n\n- RTX 5090 arch flags:\n`FLASHINFER_CUDA_ARCH_LIST=12.0a`\n\n,`TORCH_CUDA_ARCH_LIST=\"12.0+PTX\"`\n\n(already in the launch script).\n\n**Build:**\n\n- Clone with submodules and pin to the validated commit:\n\n```\ngit clone --recursive https://github.com/kvcache-ai/ktransformers && cd ktransformers\ngit checkout 7c021b4 && git submodule update --init --recursive\n```\n\n- Build the CPU-MoE extension (kt-kernel) + the bundled sglang per the official guides —\n`doc/en/install.md`\n\nand the model guide`doc/en/DeepSeek-V4-Flash.md`\n\n— using the arch flags above. **Confirm your CPU is supported*** before*converting 138 GB of weights:\n\n``` python\npython -c \"from kt_kernel.utils.amx import _HAS_AMXINT4_SUPPORT; print(_HAS_AMXINT4_SUPPORT)\"   # must print: True\n```\n\nSource weights — ** deepseek-ai/DeepSeek-V4-Flash** (MIT).\nPoint the converter at that release (its top-level quant is\n\n`fp8 e4m3`\n\n; the converter reads the *routed-expert*tensors and re-quantizes to int4 via\n\n`--input-type fp4`\n\n).**1. Convert experts to int4, NUMA-sharded** (one-time; produces ~138 GB / 44 shards):\n\n```\npython kt-kernel/scripts/convert_cpu_weights_ds4.py \\\n  -i  /path/to/dsv4_flash \\\n  -o  /path/to/dsv4_flash_rawint4 \\\n  --input-type fp4 --quant-method int4 --gpu \\\n  --cpuinfer-threads 40 --threadpool-count 2\n```\n\n**2. Launch the server** — edit the three paths at the top of [ run_dsv4_amxint4_numa_28tps.sh](https://gist.github.com/starred.atom) and run it.\nStartup is ~5–7 min (one-time load of 138 GB of CPU experts). Run it persistently; clients then connect in seconds.\n\n**3. Verify throughput + quality:**\n\n```\npython codegate.py        # executes generated code and asserts correctness (8/8 should pass)\n```\n\nThroughput: grep the server log for `gen throughput`\n\n.\n\n: this recipe uses`SGLANG_DSV4_2604_SUBMODE`\n\n`2604A`\n\n(measured); upstream docs default to`2604B`\n\n. Both work;`2604A`\n\nis what produced the numbers above on this box.**Agent serving**: OpenAI-compatible API at`http://127.0.0.1:30000/v1`\n\n, native tool-calling via`--tool-call-parser deepseekv4 --reasoning-parser deepseek-v4`\n\n. Verified end-to-end tool-call round-trips at 512K context (KV pool holds ~654K tokens at fp8).**vLLM?** Not usefully on this hardware — it has no CPU-expert-*compute*path;`--cpu-offload-gb`\n\nstreams weights over PCIe per forward pass. The speed here comes specifically from ktransformers computing int4 experts on the CPU.\n\n*Credits: DeepSeek (model, MIT); kvcache-ai/ktransformers + LMSYS/sglang (the stack that makes CPU-offload MoE fast).*", "url": "https://wpnews.pro/news/deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe", "canonical_source": "https://gist.github.com/RockmSockmJesus/30a195ccd9b62e981ec2676a99a57b7e", "published_at": "2026-07-22 14:22:05+00:00", "updated_at": "2026-07-23 00:26:33.679825+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["DeepSeek", "ktransformers", "sglang", "RTX 5090", "Xeon Gold 6138"], "alternates": {"html": "https://wpnews.pro/news/deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe", "markdown": "https://wpnews.pro/news/deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe.md", "text": "https://wpnews.pro/news/deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe.txt", "jsonld": "https://wpnews.pro/news/deepseek-v4-flash-284b-moe-at-28-tok-s-on-1x-rtx-5090-dual-xeon-gold-6138-recipe.jsonld"}}