DeepSeek-V4-Flash-0731 (UD-Q4_K_XL GGUF) stable on four RTX 3060 12GB cards. The real win here isn't just that it runs, but that I'm hitting nearly 100 tok/s during prompt processing while maintaining a massive context window of around 360k tokens.
The secret sauce is a very specific, non-intuitive tensor split and expert off strategy in llama.cpp. If you try to calculate the layout analytically, you'll probably fail because the interaction between -ncmoe
and explicit tensor overrides is weird.
The Hardware Stack #
GPU: 4× NVIDIA RTX 3060 12GB (48 GB Total VRAM)CPU: Intel Core i9-10920X (12C/24T)RAM: 128 GB DDR4-3200 (Quad-channel is key here since most of the model sits in system memory)Engine: llama.cpp build b10181
The Optimized Deployment #
To get this working, I had to push almost all non-expert tensors to GPU0 and surgically place the remaining experts on the other cards. Here is the exact command I used for the best balance of speed and stability:
llama-server \
-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \
-c 368640 \
-ncmoe 34 \
-ts 100,1,1,1 \
-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \
-ctk q8_0 \
-ctv q8_0 \
-b 2048 \
-ub 2048 \
-np 1 \
-lm none \
--threads 20 \
--flash-attn on
Performance Breakdown #
I tested this with a ~20.5k token prompt, and the results were surprisingly snappy for a setup that is heavily relying on system RAM:
Prompt processing: 99.4 tok/sText generation: 10.1 tok/sVRAM Headroom (GPU0): 671 MiB free
Key Technical Takeaways #
The most critical part of this AI workflow is how the experts are handled. By setting
-ncmoe 34
, I keep experts from blocks 0–33 in system RAM. I then manually distribute the remaining nine expert layers across GPUs 1, 2, and 3 (three layers per card). The -ts 100,1,1,1
split is aggressive; it forces the attention and KV allocations onto GPU0, leaving just enough room on the other three cards to hold those specific expert weights.
I also found that the physical microbatch size (-ub
) is the biggest performance lever. Dropping -ub
to 1024 tanked my prompt processing to about 63.4 tok/s. Boosting it to 2048 is what got me to that 100 tok/s mark, though it eats more VRAM. If you need a safer margin or a larger context (up to 524k), stick with 1024.
A few other stability notes:
KV Cache: Usingq8_0
is the sweet spot. F16 KV almost OOM'd my cards.Memory Mapping: I disabled it with-lm none
for better stability.Slots: Keep-np 1
because multiple slots multiply the KV-cache requirements and will kill your VRAM instantly.
Next Groq just bagged $350M to go all-in on the neocloud pivot →
a guide to making money with AI, with plenty of directly applicable cases.