# DeepSeek-V4-Flash (284B MoE) at ~28 tok/s on 1x RTX 5090 + dual Xeon Gold 6138 — recipe + benchmarked negatives (companion to ktransformers#2088)

> Source: <https://gist.github.com/RockmSockmJesus/30a195ccd9b62e981ec2676a99a57b7e>
> Published: 2026-07-22 14:22:05+00:00

A reproducible recipe + benchmark study — **not a new model, not a novel technique**. The core stack is stock
[kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers) + sglang. The value here is the measured
results and the **negatives**: what to run, what to skip, and why, on an accessible single-GPU + dual-Xeon box.

Companion to the upstream docs PR this study fed back:

(merged) — AMXINT4 on Skylake (no AMX), dual-Xeon tuning, and the single-stream negative results.[ktransformers#2088]

A serving recipe that runs **DeepSeek-V4-Flash** — 284B total / 13B active-parameter Mixture-of-Experts
(256 routed experts/layer, top-6, 43 layers, MLA + sparse attention, built-in MTP head) — for **single-stream
decode at ~28 tokens/sec, at 4-bit, with coding quality intact**, on a single commodity workstation. It is
OpenAI-API-compatible and drives agentic tool-calling.

The model is 142 GB at 4-bit. It does **not** fit the GPU. This recipe runs it anyway (experts computed on CPU).

| GPU | 1× RTX 5090, 32 GB (sm_120, 1.79 TB/s) |
| CPU | 2× Xeon Gold 6138 (Skylake-SP, 40 physical cores, AVX-512 — no AMX tile, no VNNI) |
| RAM | 256 GB DDR4-2400, 8/12 channels (~100 GB/s realizable), GPU on NUMA node 0 |

**CPU-offloaded experts.** Attention + a few hot experts run on the GPU; the 256 routed experts per layer are**computed on the CPU**(ktransformers/kt-kernel), in RAM. Crucially the expert*weights never cross PCIe*— only small activations do. This is what lets a 142 GB model run on a 32 GB card.**Unified stack.** sglang (scheduler, attention, MTP, OpenAI API) + ktransformers kt-kernel (CPU MoE), integrated via`kt_ep_wrapper`

. One server, one endpoint.**Low-bit.** Experts stored/computed at**int4**("AMXINT4" kernel — despite the name it needs only AVX512F+AVX512BW, so it runs on Skylake with no AMX tile). 4-bit is the near-lossless floor; coding quality holds.**Locality-optimized.** Experts are**NUMA-sharded**(one shard per socket) and read node-locally, so each socket reads from its own RAM instead of half-crossing the inter-socket link. This is where the 25 → ~28 tok/s (+14%) came from, at zero quality cost. (The AMXINT4+NUMA recipe itself is documented upstream; what's added here is the quantified gain on this specific hardware.)

Plus: **top-4 adaptive-k** (cull each token's experts from top-6 to top-4 — coding-verified), expert-deferral
CPU/GPU overlap, and frequency-based hot-expert placement.

| Config | tok/s | Quality | Notes |
|---|---|---|---|
| Original (MXFP4 + GPU experts + EAGLE) | 25 | ✓ | prior baseline |
This recipe (AMXINT4 numa, CPU-only, top-4) |
~28 |
✓ coding holds |
+14%, simpler |

Coding canary passes (is_palindrome, two_sum, binary_search, valid_parens, merge_intervals, fib,
longest_common_prefix, group_anagrams all correct; math/primes correct — see `codegate.py`

). Decode is
**DDR4-bandwidth-saturated** at this point — confirmed the ceiling.

*Re-verified 2026-07-22 on the same box: 27.9–28.0 tok/s steady-state decode (streaming measurement, time between
first and last token, prefill excluded — the same "gen throughput" the 28 refers to).*

| Lever | Result | Why it lost |
|---|---|---|
| EAGLE speculative decode | 27.2 | spec verifies a batch of draft tokens routing to different experts → multiplies the CPU expert reads that are the bottleneck |
| More GPU experts | 27.5 | enabling machinery costs more than offloading 6/256 experts saves |
| cpuinfer=80 (hyperthreads) | 2.7 | HT oversubscription thrashes the bandwidth-bound AVX-512 kernel; 40 = 1 thread/physical-core is optimal |
| Sub-4-bit expert reduction (<top-4) | breaks coding | expert-count floor is top-4 |

**Bottom line:** single-stream decode is DDR4-bandwidth-saturated at ~28 tok/s (4-bit). Going faster needs more
memory bandwidth (populate the empty DIMM channels) or sub-4-bit experts (which need a new kernel + quality care).
Post-training 2-bit/ternary breaks coding on DSV4 experts (~4× the weight error of int4); good ternary needs QAT.

The recipe is stock ktransformers + sglang — but **building that stack for the RTX 5090 (sm_120) is the actual work.**
Pin to the exact versions this was validated on; upstream flag names (`--kt-method`

, `SGLANG_DSV4_MODE`

) can drift on
`main`

.

**Hard minimums (not optional — check these before downloading 150 GB of weights):**

| Resource | Minimum | Why |
|---|---|---|
| System RAM | ~250 GB (256 GB here) |
Experts are CPU-resident: 138 GB converted weights + ~165 GB serving RSS. A 128 GB box cannot load it. |
| Free disk | ~300 GB | ~150 GB source download + 138 GB converted shards |
| CPU | x86-64 with AVX512F + AVX512BW |
The "AMXINT4" int4 kernel needs these — but NOT the AMX tile or VNNI (verified on Skylake-SP that has neither). |
| GPU | CUDA GPU, ~24 GB VRAM used at 8K ctx | Holds attention + KV only; experts stay on CPU. sm_120 for the 5090. |

Dual-socket NUMA is what buys the +14% locality win, but a single socket works (the recipe just collapses to one shard).

**Validated environment (pin these):**

**ktransformers @**—`7c021b430c36a408032c20bbf3833dc1bce6efa4`

*verified stock: this recipe runs with*(confirmed clean working tree; the top-4 override and AMXINT4 path are all upstream).**zero** local patches- sglang submodule @
`1e098a77ba395dc1a5f2dcbdf57bdb188e84bcee`

**torch**`2.9.1+cu128`

, CUDA`12.8`

- RTX 5090 arch flags:
`FLASHINFER_CUDA_ARCH_LIST=12.0a`

,`TORCH_CUDA_ARCH_LIST="12.0+PTX"`

(already in the launch script).

**Build:**

- Clone with submodules and pin to the validated commit:

```
git clone --recursive https://github.com/kvcache-ai/ktransformers && cd ktransformers
git checkout 7c021b4 && git submodule update --init --recursive
```

- Build the CPU-MoE extension (kt-kernel) + the bundled sglang per the official guides —
`doc/en/install.md`

and the model guide`doc/en/DeepSeek-V4-Flash.md`

— using the arch flags above. **Confirm your CPU is supported*** before*converting 138 GB of weights:

``` python
python -c "from kt_kernel.utils.amx import _HAS_AMXINT4_SUPPORT; print(_HAS_AMXINT4_SUPPORT)"   # must print: True
```

Source weights — ** deepseek-ai/DeepSeek-V4-Flash** (MIT).
Point the converter at that release (its top-level quant is

`fp8 e4m3`

; the converter reads the *routed-expert*tensors and re-quantizes to int4 via

`--input-type fp4`

).**1. Convert experts to int4, NUMA-sharded** (one-time; produces ~138 GB / 44 shards):

```
python kt-kernel/scripts/convert_cpu_weights_ds4.py \
  -i  /path/to/dsv4_flash \
  -o  /path/to/dsv4_flash_rawint4 \
  --input-type fp4 --quant-method int4 --gpu \
  --cpuinfer-threads 40 --threadpool-count 2
```

**2. Launch the server** — edit the three paths at the top of [ run_dsv4_amxint4_numa_28tps.sh](https://gist.github.com/starred.atom) and run it.
Startup is ~5–7 min (one-time load of 138 GB of CPU experts). Run it persistently; clients then connect in seconds.

**3. Verify throughput + quality:**

```
python codegate.py        # executes generated code and asserts correctness (8/8 should pass)
```

Throughput: grep the server log for `gen throughput`

.

: this recipe uses`SGLANG_DSV4_2604_SUBMODE`

`2604A`

(measured); upstream docs default to`2604B`

. Both work;`2604A`

is what produced the numbers above on this box.**Agent serving**: OpenAI-compatible API at`http://127.0.0.1:30000/v1`

, native tool-calling via`--tool-call-parser deepseekv4 --reasoning-parser deepseek-v4`

. Verified end-to-end tool-call round-trips at 512K context (KV pool holds ~654K tokens at fp8).**vLLM?** Not usefully on this hardware — it has no CPU-expert-*compute*path;`--cpu-offload-gb`

streams weights over PCIe per forward pass. The speed here comes specifically from ktransformers computing int4 experts on the CPU.

*Credits: DeepSeek (model, MIT); kvcache-ai/ktransformers + LMSYS/sglang (the stack that makes CPU-offload MoE fast).*