# Four RTX 3060s can actually push 100 tok/s prompt processing on

> Source: <https://promptcube3.com/en/threads/6815/>
> Published: 2026-08-18 17:18:26+00:00

# Four RTX 3060s can actually push 100 tok/s prompt processing on

[DeepSeek](/en/tags/deepseek/)-V4-Flash-0731 (UD-Q4_K_XL GGUF) stable on four RTX 3060 12GB cards. The real win here isn't just that it runs, but that I'm hitting nearly 100 tok/s during prompt processing while maintaining a massive context window of around 360k tokens.

The secret sauce is a very specific, non-intuitive tensor split and expert offloading strategy in llama.cpp. If you try to calculate the layout analytically, you'll probably fail because the interaction between `-ncmoe`

and explicit tensor overrides is weird.

## The Hardware Stack

**GPU:** 4× NVIDIA RTX 3060 12GB (48 GB Total VRAM)**CPU:** Intel Core i9-10920X (12C/24T)**RAM:** 128 GB DDR4-3200 (Quad-channel is key here since most of the model sits in system memory)**Engine:** llama.cpp build b10181

## The Optimized Deployment

To get this working, I had to push almost all non-expert tensors to GPU0 and surgically place the remaining experts on the other cards. Here is the exact command I used for the best balance of speed and stability:

```
llama-server \
-m DeepSeek-V4-Flash-0731-UD-Q4_K_XL-00001-of-00005.gguf \
-c 368640 \
-ncmoe 34 \
-ts 100,1,1,1 \
-ot 'blk.(3[4-6]).ffn_.*_exps=CUDA1,blk.(3[7-9]).ffn_.*_exps=CUDA2,blk.(4[0-2]).ffn_.*_exps=CUDA3' \
-ctk q8_0 \
-ctv q8_0 \
-b 2048 \
-ub 2048 \
-np 1 \
-lm none \
--threads 20 \
--flash-attn on
```

## Performance Breakdown

I tested this with a ~20.5k token prompt, and the results were surprisingly snappy for a setup that is heavily relying on system RAM:

**Prompt processing:** 99.4 tok/s**Text generation:** 10.1 tok/s**VRAM Headroom (GPU0):** 671 MiB free

## Key Technical Takeaways

The most critical part of this AI workflow is how the experts are handled. By setting

`-ncmoe 34`

, I keep experts from blocks 0–33 in system RAM. I then manually distribute the remaining nine expert layers across GPUs 1, 2, and 3 (three layers per card). The `-ts 100,1,1,1`

split is aggressive; it forces the attention and KV allocations onto GPU0, leaving just enough room on the other three cards to hold those specific expert weights.

I also found that the physical microbatch size (`-ub`

) is the biggest performance lever. Dropping `-ub`

to 1024 tanked my prompt processing to about 63.4 tok/s. Boosting it to 2048 is what got me to that 100 tok/s mark, though it eats more VRAM. If you need a safer margin or a larger context (up to 524k), stick with 1024.

A few other stability notes:

**KV Cache:** Using`q8_0`

is the sweet spot. F16 KV almost OOM'd my cards.**Memory Mapping:** I disabled it with`-lm none`

for better stability.**Slots:** Keep`-np 1`

because multiple slots multiply the KV-cache requirements and will kill your VRAM instantly.

[Next Groq just bagged $350M to go all-in on the neocloud pivot →](/en/threads/6727/)

[a guide to making money with AI](https://tanyan888.com/), with plenty of directly applicable cases.