{"slug": "four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on", "title": "Four RTX 3060s can actually push 100 tok/s prompt processing on", "summary": "A developer reports achieving 99.4 tok/s prompt processing and 10.1 tok/s text generation with DeepSeek-V4-Flash-0731 (UD-Q4_K_XL GGUF) on four NVIDIA RTX 3060 12GB cards using llama.cpp build b10181, maintaining a 368,640-token context. The setup relies on a specific tensor split and expert offloading strategy, with the physical microbatch size identified as the biggest performance lever.", "body_md": "# Four RTX 3060s can actually push 100 tok/s prompt processing on\n\n[DeepSeek](/en/tags/deepseek/)-V4-Flash-0731 (UD-Q4_K_XL GGUF) stable on four RTX 3060 12GB cards. The real win here isn't just that it runs, but that I'm hitting nearly 100 tok/s during prompt processing while maintaining a massive context window of around 360k tokens.\n\nThe secret sauce is a very specific, non-intuitive tensor split and expert offloading strategy in llama.cpp. If you try to calculate the layout analytically, you'll probably fail because the interaction between `-ncmoe`\n\nand explicit tensor overrides is weird.\n\n## The Hardware Stack\n\n**GPU:** 4× NVIDIA RTX 3060 12GB (48 GB Total VRAM)**CPU:** Intel Core i9-10920X (12C/24T)**RAM:** 128 GB DDR4-3200 (Quad-channel is key here since most of the model sits in system memory)**Engine:** llama.cpp build b10181\n\n## The Optimized Deployment\n\nTo get this working, I had to push almost all non-expert tensors to GPU0 and surgically place the remaining experts on the other cards. Here is the exact command I used for the best balance of speed and stability:\n\n```\nllama-server \\\n-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \\\n-c 368640 \\\n-ncmoe 34 \\\n-ts 100,1,1,1 \\\n-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \\\n-ctk q8_0 \\\n-ctv q8_0 \\\n-b 2048 \\\n-ub 2048 \\\n-np 1 \\\n-lm none \\\n--threads 20 \\\n--flash-attn on\n```\n\n## Performance Breakdown\n\nI tested this with a ~20.5k token prompt, and the results were surprisingly snappy for a setup that is heavily relying on system RAM:\n\n**Prompt processing:** 99.4 tok/s**Text generation:** 10.1 tok/s**VRAM Headroom (GPU0):** 671 MiB free\n\n## Key Technical Takeaways\n\nThe most critical part of this AI workflow is how the experts are handled. By setting\n\n`-ncmoe 34`\n\n, I keep experts from blocks 0–33 in system RAM. I then manually distribute the remaining nine expert layers across GPUs 1, 2, and 3 (three layers per card). The `-ts 100,1,1,1`\n\nsplit is aggressive; it forces the attention and KV allocations onto GPU0, leaving just enough room on the other three cards to hold those specific expert weights.\n\nI also found that the physical microbatch size (`-ub`\n\n) is the biggest performance lever. Dropping `-ub`\n\nto 1024 tanked my prompt processing to about 63.4 tok/s. Boosting it to 2048 is what got me to that 100 tok/s mark, though it eats more VRAM. If you need a safer margin or a larger context (up to 524k), stick with 1024.\n\nA few other stability notes:\n\n**KV Cache:** Using`q8_0`\n\nis the sweet spot. F16 KV almost OOM'd my cards.**Memory Mapping:** I disabled it with`-lm none`\n\nfor better stability.**Slots:** Keep`-np 1`\n\nbecause multiple slots multiply the KV-cache requirements and will kill your VRAM instantly.\n\n[Next Groq just bagged $350M to go all-in on the neocloud pivot →](/en/threads/6727/)\n\n[a guide to making money with AI](https://tanyan888.com/), with plenty of directly applicable cases.", "url": "https://wpnews.pro/news/four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on", "canonical_source": "https://promptcube3.com/en/threads/6815/", "published_at": "2026-08-18 17:18:26+00:00", "updated_at": "2026-08-18 17:42:26.074568+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-infrastructure", "ai-tools"], "entities": ["DeepSeek-V4-Flash-0731", "NVIDIA RTX 3060", "llama.cpp", "Intel Core i9-10920X"], "alternates": {"html": "https://wpnews.pro/news/four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on", "markdown": "https://wpnews.pro/news/four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on.md", "text": "https://wpnews.pro/news/four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on.txt", "jsonld": "https://wpnews.pro/news/four-rtx-3060s-can-actually-push-100-tok-s-prompt-processing-on.jsonld"}}