Four RTX 3060s can actually push 100 tok/s prompt processing on

A developer reports achieving 99.4 tok/s prompt processing and 10.1 tok/s text generation with DeepSeek-V4-Flash-0731 (UD-Q4_K_XL GGUF) on four NVIDIA RTX 3060 12GB cards using llama.cpp build b10181, maintaining a 368,640-token context. The setup relies on a specific tensor split and expert offloading strategy, with the physical microbatch size identified as the biggest performance lever.

Four RTX 3060s can actually push 100 tok/s prompt processing on DeepSeek /en/tags/deepseek/ -V4-Flash-0731 UD-Q4 K XL GGUF stable on four RTX 3060 12GB cards. The real win here isn't just that it runs, but that I'm hitting nearly 100 tok/s during prompt processing while maintaining a massive context window of around 360k tokens. The secret sauce is a very specific, non-intuitive tensor split and expert offloading strategy in llama.cpp. If you try to calculate the layout analytically, you'll probably fail because the interaction between -ncmoe and explicit tensor overrides is weird. The Hardware Stack GPU: 4× NVIDIA RTX 3060 12GB 48 GB Total VRAM CPU: Intel Core i9-10920X 12C/24T RAM: 128 GB DDR4-3200 Quad-channel is key here since most of the model sits in system memory Engine: llama.cpp build b10181 The Optimized Deployment To get this working, I had to push almost all non-expert tensors to GPU0 and surgically place the remaining experts on the other cards. Here is the exact command I used for the best balance of speed and stability: llama-server \ -m DeepSeek-V4-Flash-0731-UD-Q4 K XL-00001-of-00005.gguf \ -c 368640 \ -ncmoe 34 \ -ts 100,1,1,1 \ -ot 'blk. 3 4-6 .ffn . exps=CUDA1,blk. 3 7-9 .ffn . exps=CUDA2,blk. 4 0-2 .ffn . exps=CUDA3' \ -ctk q8 0 \ -ctv q8 0 \ -b 2048 \ -ub 2048 \ -np 1 \ -lm none \ --threads 20 \ --flash-attn on Performance Breakdown I tested this with a ~20.5k token prompt, and the results were surprisingly snappy for a setup that is heavily relying on system RAM: Prompt processing: 99.4 tok/s Text generation: 10.1 tok/s VRAM Headroom GPU0 : 671 MiB free Key Technical Takeaways The most critical part of this AI workflow is how the experts are handled. By setting -ncmoe 34 , I keep experts from blocks 0–33 in system RAM. I then manually distribute the remaining nine expert layers across GPUs 1, 2, and 3 three layers per card . The -ts 100,1,1,1 split is aggressive; it forces the attention and KV allocations onto GPU0, leaving just enough room on the other three cards to hold those specific expert weights. I also found that the physical microbatch size -ub is the biggest performance lever. Dropping -ub to 1024 tanked my prompt processing to about 63.4 tok/s. Boosting it to 2048 is what got me to that 100 tok/s mark, though it eats more VRAM. If you need a safer margin or a larger context up to 524k , stick with 1024. A few other stability notes: KV Cache: Using q8 0 is the sweet spot. F16 KV almost OOM'd my cards. Memory Mapping: I disabled it with -lm none for better stability. Slots: Keep -np 1 because multiple slots multiply the KV-cache requirements and will kill your VRAM instantly. Next Groq just bagged $350M to go all-in on the neocloud pivot → /en/threads/6727/ a guide to making money with AI https://tanyan888.com/ , with plenty of directly applicable cases.