A reproducible recipe + benchmark study — not a new model, not a novel technique. The core stack is stock kvcache-ai/ktransformers + sglang. The value here is the measured results and the negatives: what to run, what to skip, and why, on an accessible single-GPU + dual-Xeon box.
Companion to the upstream docs PR this study fed back:
(merged) — AMXINT4 on Skylake (no AMX), dual-Xeon tuning, and the single-stream negative results.[ktransformers#2088]
A serving recipe that runs DeepSeek-V4-Flash — 284B total / 13B active-parameter Mixture-of-Experts (256 routed experts/layer, top-6, 43 layers, MLA + sparse attention, built-in MTP head) — for single-stream decode at ~28 tokens/sec, at 4-bit, with coding quality intact, on a single commodity workstation. It is OpenAI-API-compatible and drives agentic tool-calling.
The model is 142 GB at 4-bit. It does not fit the GPU. This recipe runs it anyway (experts computed on CPU).
| GPU | 1× RTX 5090, 32 GB (sm_120, 1.79 TB/s) | | CPU | 2× Xeon Gold 6138 (Skylake-SP, 40 physical cores, AVX-512 — no AMX tile, no VNNI) | | RAM | 256 GB DDR4-2400, 8/12 channels (~100 GB/s realizable), GPU on NUMA node 0 |
CPU-offloaded experts. Attention + a few hot experts run on the GPU; the 256 routed experts per layer arecomputed on the CPU(ktransformers/kt-kernel), in RAM. Crucially the expertweights never cross PCIe— only small activations do. This is what lets a 142 GB model run on a 32 GB card.Unified stack. sglang (scheduler, attention, MTP, OpenAI API) + ktransformers kt-kernel (CPU MoE), integrated viakt_ep_wrapper
. One server, one endpoint.Low-bit. Experts stored/computed atint4("AMXINT4" kernel — despite the name it needs only AVX512F+AVX512BW, so it runs on Skylake with no AMX tile). 4-bit is the near-lossless floor; coding quality holds.Locality-optimized. Experts areNUMA-sharded(one shard per socket) and read node-locally, so each socket reads from its own RAM instead of half-crossing the inter-socket link. This is where the 25 → ~28 tok/s (+14%) came from, at zero quality cost. (The AMXINT4+NUMA recipe itself is documented upstream; what's added here is the quantified gain on this specific hardware.)
Plus: top-4 adaptive-k (cull each token's experts from top-6 to top-4 — coding-verified), expert-deferral CPU/GPU overlap, and frequency-based hot-expert placement.
| Config | tok/s | Quality | Notes |
|---|---|---|---|
| Original (MXFP4 + GPU experts + EAGLE) | 25 | ✓ | prior baseline |
| This recipe (AMXINT4 numa, CPU-only, top-4) | |||
| ~28 | |||
| ✓ coding holds | |||
| +14%, simpler |
Coding canary passes (is_palindrome, two_sum, binary_search, valid_parens, merge_intervals, fib,
longest_common_prefix, group_anagrams all correct; math/primes correct — see codegate.py
). Decode is DDR4-bandwidth-saturated at this point — confirmed the ceiling.
Re-verified 2026-07-22 on the same box: 27.9–28.0 tok/s steady-state decode (streaming measurement, time between first and last token, prefill excluded — the same "gen throughput" the 28 refers to).
| Lever | Result | Why it lost |
|---|---|---|
| EAGLE speculative decode | 27.2 | spec verifies a batch of draft tokens routing to different experts → multiplies the CPU expert reads that are the bottleneck |
| More GPU experts | 27.5 | enabling machinery costs more than off 6/256 experts saves |
| cpuinfer=80 (hyperthreads) | 2.7 | HT oversubscription thrashes the bandwidth-bound AVX-512 kernel; 40 = 1 thread/physical-core is optimal |
| Sub-4-bit expert reduction (<top-4) | breaks coding | expert-count floor is top-4 |
Bottom line: single-stream decode is DDR4-bandwidth-saturated at ~28 tok/s (4-bit). Going faster needs more memory bandwidth (populate the empty DIMM channels) or sub-4-bit experts (which need a new kernel + quality care). Post-training 2-bit/ternary breaks coding on DSV4 experts (~4× the weight error of int4); good ternary needs QAT.
The recipe is stock ktransformers + sglang — but building that stack for the RTX 5090 (sm_120) is the actual work.
Pin to the exact versions this was validated on; upstream flag names (--kt-method
, SGLANG_DSV4_MODE
) can drift on
main
.
Hard minimums (not optional — check these before down 150 GB of weights):
| Resource | Minimum | Why |
|---|---|---|
| System RAM | ~250 GB (256 GB here) | |
| Experts are CPU-resident: 138 GB converted weights + ~165 GB serving RSS. A 128 GB box cannot load it. | ||
| Free disk | ~300 GB | ~150 GB source download + 138 GB converted shards |
| CPU | x86-64 with AVX512F + AVX512BW | |
| The "AMXINT4" int4 kernel needs these — but NOT the AMX tile or VNNI (verified on Skylake-SP that has neither). | ||
| GPU | CUDA GPU, ~24 GB VRAM used at 8K ctx | Holds attention + KV only; experts stay on CPU. sm_120 for the 5090. |
Dual-socket NUMA is what buys the +14% locality win, but a single socket works (the recipe just collapses to one shard).
Validated environment (pin these):
ktransformers @—7c021b430c36a408032c20bbf3833dc1bce6efa4
verified stock: this recipe runs with(confirmed clean working tree; the top-4 override and AMXINT4 path are all upstream).zero local patches- sglang submodule @
1e098a77ba395dc1a5f2dcbdf57bdb188e84bcee
torch2.9.1+cu128
, CUDA12.8
- RTX 5090 arch flags:
FLASHINFER_CUDA_ARCH_LIST=12.0a
,TORCH_CUDA_ARCH_LIST="12.0+PTX"
(already in the launch script).
Build:
- Clone with submodules and pin to the validated commit:
git clone --recursive https://github.com/kvcache-ai/ktransformers && cd ktransformers
git checkout 7c021b4 && git submodule update --init --recursive
- Build the CPU-MoE extension (kt-kernel) + the bundled sglang per the official guides —
doc/en/install.md
and the model guidedoc/en/DeepSeek-V4-Flash.md
— using the arch flags above. Confirm your CPU is supported* before*converting 138 GB of weights:
python -c "from kt_kernel.utils.amx import _HAS_AMXINT4_SUPPORT; print(_HAS_AMXINT4_SUPPORT)" # must print: True
Source weights — ** deepseek-ai/DeepSeek-V4-Flash** (MIT). Point the converter at that release (its top-level quant is
fp8 e4m3
; the converter reads the routed-experttensors and re-quantizes to int4 via
--input-type fp4
).1. Convert experts to int4, NUMA-sharded (one-time; produces ~138 GB / 44 shards):
python kt-kernel/scripts/convert_cpu_weights_ds4.py \
-i /path/to/dsv4_flash \
-o /path/to/dsv4_flash_rawint4 \
--input-type fp4 --quant-method int4 --gpu \
--cpuinfer-threads 40 --threadpool-count 2
2. Launch the server — edit the three paths at the top of run_dsv4_amxint4_numa_28tps.sh and run it. Startup is ~5–7 min (one-time load of 138 GB of CPU experts). Run it persistently; clients then connect in seconds.
3. Verify throughput + quality:
python codegate.py # executes generated code and asserts correctness (8/8 should pass)
Throughput: grep the server log for gen throughput
.
: this recipe usesSGLANG_DSV4_2604_SUBMODE
2604A
(measured); upstream docs default to2604B
. Both work;2604A
is what produced the numbers above on this box.Agent serving: OpenAI-compatible API athttp://127.0.0.1:30000/v1
, native tool-calling via--tool-call-parser deepseekv4 --reasoning-parser deepseek-v4
. Verified end-to-end tool-call round-trips at 512K context (KV pool holds ~654K tokens at fp8).vLLM? Not usefully on this hardware — it has no CPU-expert-computepath;--cpu-offload-gb
streams weights over PCIe per forward pass. The speed here comes specifically from ktransformers computing int4 experts on the CPU.
Credits: DeepSeek (model, MIT); kvcache-ai/ktransformers + LMSYS/sglang (the stack that makes CPU-offload MoE fast).