KTransformers Review: Trading GPU VRAM for CPU RAM and NUMA Complexity

We put KTransformers on our evaluation list because “buy enough GPUs to hold every expert” is an expensive answer to a sparse-model problem. A large mixture-of-experts model activates only a subset of its experts for each token, but its inactive weights still need somewhere to live.

Published DeepSeek-V3 Memory RequirementsSingle socket DRAM 382G DRAMDual socket DRAM 1T DRAMV0.3 preview DRAM 644 GB DRAMSingle-socket VRAM minimum 14 GB VRAM

The published configurations trade GPU memory for substantial CPU RAM; these capacity figures are not interchangeable measured memory peaks.

KTransformers makes a different allocation: keep routed experts in CPU memory and execute their computation on the CPU, while GPU-resident components handle attention and other compute-intensive work. The SGLang integration can also keep selected hot experts on GPUs.

That is not ordinary layer off. We are evaluating a heterogeneous execution engine, with CPU kernels, GPU kernels, memory placement, and synchronization all contributing to the result.

Our evidence boundary matters: we did not execute a GPU inference benchmark for this review. We lacked the required hardware and model weights. The performance numbers below are explicitly identified as published measurements; commands are reproduction instructions, not a fabricated lab transcript. We can assess the design and deployment requirements, but we cannot honestly supply independent RAM-pressure curves, build timings, or a measured vLLM performance gap.

Our buying question is still concrete: can a RAM-rich node deliver an acceptable large-model experience for less money than a GPU-resident deployment?

The answer looks promising for low-concurrency workloads. It is much less settled for latency-sensitive public serving.

We separate three mechanisms before making that decision:

We would not combine these into one speedup number. Reducing DeepSeek’s activated experts from eight to six changes the inference configuration. Expert deferral is a separate mechanism with its own quality implications.

We anchor our historical setup review and throughput analysis in the DeepSeek R1/V3 configuration. We use the LMSYS integration article to examine the CPU-kernel and SGLang architecture. Its linked SOSP paper page is the primary research reference for the hybrid design; we do not treat it as an independently reproduced result.

We would reproduce two tracks separately: the historical native GGUF setup and the newer SGLang/kt-kernel path. Mixing their commands, formats, and performance tables would produce an irreproducible benchmark.

Our target for reproducing the historical setup is a Linux node with two Xeon Gold 6454S sockets, 32 cores per socket, DDR5 memory, and a 24 GB RTX 4090-class GPU. Those are reference conditions, not hardware we tested.

Before down hundreds of gigabytes, we would capture:

mkdir -p artifacts
lscpu > artifacts/lscpu.txt
numactl --hardware > artifacts/numa.txt
nvidia-smi -q > artifacts/gpu.txt
free -b > artifacts/memory.txt
python --version > artifacts/python.txt 2>&1
gcc --version > artifacts/gcc.txt
cmake --version > artifacts/cmake.txt

We care about physical cores, memory channels populated per socket, GPU-to-NUMA affinity, and available memory—not merely the total RAM printed on a sales quote.

For a lightweight installation check, the following block is runnable in an environment with kt-kernel installed:

python - <<'PY'
import json
import kt_kernel
from kt_kernel import kt_kernel_ext

engine = kt_kernel_ext.CPUInfer(4)
print(json.dumps({
    "version": getattr(kt_kernel, "__version__", "unknown"),
    "cpu_variant": getattr(kt_kernel, "__cpu_variant__", "unknown"),
    "cuda_stream_api_present": hasattr(engine, "submit_with_cuda_stream"),
    "inference_benchmark_executed": False
}, indent=2))
PY

Illustrative output only—not observed output:

{
  "version": "example-installed-version",
  "cpu_variant": "amx",
  "cuda_stream_api_present": true,
  "inference_benchmark_executed": false
}

That check establishes importability and an exposed interface. It does not establish GPU functionality, correct model , or inference performance.

For our reproduction of the 2025 integration, we would use Python 3.11, CUDA 12.1 or later, GCC/G++ 11 or later, and CMake 3.25 or later.

For the newer kt-kernel installation path, we would use prebuilt wheels and the sglang-kt fork rather than assume the stock sglang package supports the same integration:

python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install kt-kernel sglang-kt
python -m pip freeze > artifacts/requirements-resolved.txt

This is a bootstrap, not a pinned benchmark environment. Before collecting results, we would freeze exact package versions, archive wheel hashes, and record the repository commit containing the conversion script. We would keep pure-GPU SGLang and vLLM in separate environments.

We distinguish two preparation paths.

For our historical native reproduction, we would supply a model directory plus a directory containing Q4_K_M GGUF weights. For the BF16 accuracy-check path, we would convert the source weights to GGUF through llama.cpp. Existing GGUF downloads avoid local quantization work, but still require verification of model identity and format.

For the newer AMX path, we need CPU weights converted into the kernel’s INT4 or INT8 layout, in addition to the GPU-side model:

python scripts/convert_cpu_weights.py \
  --input-path /models/DeepSeek-V3 \
  --input-type fp8 \
  --output /models/DeepSeek-V3-CPU-INT4 \
  --quant-method int4

The input type must match the actual source weights. We would measure conversion elapsed time, peak memory, temporary disk consumption, and final directory size. None of those measurements is available here.

A conservative starting configuration for the newer integration is:

python -m sglang.launch_server \
  --host 127.0.0.1 \
  --port 30000 \
  --model /models/DeepSeek-V3 \
  --kt-method AMXINT4 \
  --kt-weight-path /models/DeepSeek-V3-CPU-INT4 \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 0 \
  --kt-max-deferred-experts-per-token 0 \
  --tensor-parallel-size 1 \
  --attention-backend triton \
  --mem-fraction-static 0.80 \
  --chunked-prefill-size 256 \
  --max-running-requests 1 \
  --max-total-tokens 8192

We would verify every flag against the pinned build’s --help. This is an evaluation starting point, not a guarantee that a particular model artifact fits.

We start without GPU-resident routed experts or deferral to isolate the basic hybrid path. Then we change one variable at a time.

We would sweep prompt lengths around 1K, 2K, 4K, and 8K, request 300 output tokens, and record actual token counts. After warm-up, we would repeat each cell and retain:

We would add concurrency only after establishing the single-request baseline. A rising aggregate token rate can conceal a deteriorating interactive experience.

Our first review blocker was version ambiguity, not a measured runtime failure. The historical launch examples use --kt-amx-weight-path and --kt-amx-method; newer instructions use --kt-weight-path and --kt-method. We would not silently translate an old command and call it a reproduction.

For our historical Q4_K_M evaluation, we use reference memory figures of approximately 382G DRAM and at least 14 GB VRAM for the single-socket configuration and 1T DRAM for the dual-socket configuration. We keep the V0.3 preview's 644 GB DRAM figure separate: that setup uses BF16 source weights quantized online to INT8 for the CPU and INT4 for the GPU.

We would not interpret those entries as interchangeable measured resident-set peaks. In our capacity planning for the historical dual-socket path, we account for critical matrices copied across NUMA nodes, which add memory and overhead.

Our capacity gate therefore includes weights, converted layouts, replication, runtime allocations, and operating-system headroom. A node with enough total RAM may still lack enough memory local to the selected socket.

We also cannot infer sustained RAM pressure from those capacity figures. Swap behavior, bandwidth saturation, peak memory, and the OOM boundary remain unmeasured.

The historical single-socket command binds both execution and memory:

numactl -N 1 -m 1 python ./ktransformers/local_chat.py \
  --model_path /models/DeepSeek-V3 \
  --gguf_path /models/DeepSeek-V3-Q4_K_M \
  --prompt_file /data/prompt.txt \
  --cpu_infer 33 \
  --max_new_tokens 1000

Node 1 is not universally correct. We would inspect topology first.

For the historical dual-socket native build, USE_NUMA=1 must be set before installation; an existing installation needs rebuilding. The newer integration exposes NUMA-oriented thread pools instead.

We would compare socket-local execution, unmanaged placement, and the supported two-pool configuration. We would not force interleaving and assume it reproduces the kernel’s placement strategy.

In our NUMA evaluation, we use the up-to-63% decoding improvement for NUMA-aware tensor parallelism on dual-socket servers as a published reference point, not an independently measured result or an uplift guaranteed on our next server.

For our BF16 reproduction plan, we flag older g++ and assembler versions as compatibility risks, particularly on Ubuntu 20 or earlier. We would still install the missing KTransformers package and any other required packages inside the provided development container.

Prebuilt wheels reduce compilation exposure, but do not prove that the complete serving stack is compatible. We would verify the selected CPU backend and archive the build log before starting expensive downloads.

We also would not use the benchmark file’s accuracy table as a serving benchmark. It uses 1,000 shuffled MMLU examples, skips few-shot prompting, and sets temperature to 0.6. We would preserve those conditions for reproduction, then add domain-specific quality checks.

Finally, we would evaluate six-expert activation and expert deferral separately. The latter has a published up-to-1.45× decoding improvement with reported accuracy variation below 0.5%; that is not permission to assume unchanged behavior on our workload.

We find the historical eight-expert results more useful than the headline maximum because they avoid crediting reduced expert activation as a free optimization.

Published reference measurements—not Effloow benchmark runs:

For the eight-L20 workload, we keep 227.85 total tok/s separate from 87.58 output tok/s. The same result includes mean inter-token latency of 431.61 ms, median 299.18 ms, and P99 1,935.13 ms.

Those tails matter. We would not describe that workload as uniformly smooth streaming merely because total throughput exceeds 220 tok/s.

We also cannot turn the tutorial’s llama.cpp comparison into a vLLM comparison. A pure-GPU deployment requires enough aggregate VRAM for the chosen weights, KV cache, and runtime workspace. A smaller model that fits one GPU answers a different question.

For a fair purchase decision, we would run matched model revisions, quantization settings, prompts, output lengths, and quality constraints. We would report both saturation throughput and throughput that meets the latency SLO.

We use accepted output tokens, not combined input/output throughput:

cost per million output tokens =
    hourly node cost × 1,000,000
    / (3,600 × sustained output tokens/s × productive utilization)

Consider an illustrative financial scenario, not measured pricing or performance:

The resulting costs are approximately $111.11 and $55.56 per million output tokens, respectively.

The cheaper node loses on token economics because throughput falls faster than price. At those assumed throughputs and equal utilization, the hybrid node breaks even at $1/hour and must cost less than that to beat an $8/hour GPU node.

Existing hardware changes the calculation, but does not eliminate deployment costs. We would include power, depreciation, conversion storage, failed builds, and operator time. For hosted APIs, we would compare the actual input/output mix, caching, privacy requirements, and service guarantees—not output price alone.

Without a hardware run, we could not independently verify inference performance, peak memory use, conversion costs, or a cost advantage over a matched GPU-resident deployment.

We would shortlist KTransformers as a capacity-enabling architecture, not declare it a universal serving replacement.

Its strongest proposition is specific: use substantial CPU RAM and optimized expert execution to make a large sparse model practical with less GPU memory. The published results establish a credible reference point. They do not establish our production latency envelope.

We would deploy a controlled pilot if:

We would hold off if:

Our final acceptance gate remains a real hardware run: sustained throughput, latency distributions, memory-limit sweeps, NUMA isolation, conversion cost, and a matched GPU-resident baseline. Until then, we would approve evaluation spending—not promise production savings.

For adjacent serving options, explore our tools collection. If the decision is whether to buy more GPUs or reuse a large-memory server, talk to us about an infrastructure evaluation. The right answer is the configuration that meets quality and latency requirements at the lowest fully accounted cost—not the one with the smallest VRAM headline.