Does a Second GPU Make Your Local LLM Faster? Layer Split vs Tensor Parallel, Explained

Llama.cpp's default layer-split mode gives a dual-GPU setup the combined VRAM of both cards but runs single-stream generation at roughly the speed of one card, while tensor parallel can nearly double single-stream throughput at the cost of per-layer synchronization traffic, according to an analysis of the project's multi-GPU guide and published memory-bandwidth figures. A Llama 70B at Q4_K_M holds about 42GB of weights, and at the RTX 3090's 936GB/s memory bandwidth the layer-split ceiling across two cards is about 22 tokens per second, with r/LocalLLaMA owners reporting 17.9 tokens per second at short context falling to 7.9 at 32K. Layer split moves only about 16KB per generated token between cards, so a PCIe 3.0 x1 riser suffices, whereas tensor parallel makes the inter-card link matter.

Editor checklist: every measured number below is attributed and linked; the ceilings are arithmetic from published memory bandwidth, labelled as such. We have not run these configurations. Delete this box before publishing. The most common question from people pricing a second RTX 3090 is whether it will make their model faster. The answer depends on a setting most of them have never touched. llama.cpp can split a model across GPUs in two fundamentally different ways, and on a single chat stream one of them adds memory and almost no speed, while the other can roughly double throughput at the cost of constant chatter between the cards. Which one you get decides whether PCIe lanes, risers and NVLink bridges matter at all. This is how both work, what the measurements say, and what to buy. The short answer - Layer split llama.cpp's default gives you the combined memory of both cards. For one user generating one reply, it runs at roughly the speed of a single card, because the cards take turns. - Tensor parallel makes both cards work on every token at once. It can nearly double single-stream speed on two cards, but every layer ends in a synchronisation, so the link between the cards starts to matter. - Serving many users changes the maths again: batching hides the synchronisation cost, and multi-GPU throughput scales well either way. Layer split: two cards taking turns In llama.cpp's default mode, --split-mode layer , each GPU holds a contiguous block of the model's layers along with the KV cache for those layers. The project's multi-GPU guide https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md?ref=vettedconsumer.com calls this pipeline parallelism and describes its trade plainly: it "minimizes data transfers between GPUs but requires many tokens to scale well." The idea goes back to GPipe Huang et al., 2018, arXiv:1811.06965 https://arxiv.org/abs/1811.06965?ref=vettedconsumer.com , which placed different sub-sequences of layers on separate accelerators and kept them all busy by feeding the pipeline many micro-batches at once. Generation is the case where there are no micro-batches. To produce one token, card one runs its 40 layers, hands a single activation vector to card two, and waits. Card two runs its 40 layers, and the token comes out. At any instant one card is working and the other is idle. Since decode speed on a GPU is set by how fast it can read its weights the bandwidth-not-TFLOPS rule https://vettedconsumer.com/bandwidth-not-tflops-what-sets-your-local-llm-speed-and-why-the-newest-card-isnt-always-fastest/ , two cards reading half the model each, one after the other, take the same time as one imaginary card with twice the memory reading all of it. The numbers line up. A Llama 70B at Q4 K M is about 42GB of weights 70 billion × 4.8 bits ÷ 8 . A 3090 reads memory at 936GB/s, so the ceiling for layer-split generation across two of them is 936 ÷ 42, about 22 tokens per second. Owners on r/LocalLLaMA https://www.reddit.com/r/LocalLLaMA/comments/1he2v2n/?ref=vettedconsumer.com report 17.9 tokens per second at short context on a dual-3090 box, falling to 7.9 at 32K. That is a healthy fraction of the ceiling, and it is also exactly what one 48GB card with 3090 bandwidth would do. The second card bought the capacity to load the model, not speed. Our earlier two 3090s versus one 5090 https://vettedconsumer.com/two-used-rtx-3090-vs-one-rtx-5090-for-local-llms/ comparison is really this effect in buying-guide form. Layer split has a large practical upside: the cards barely talk. What crosses between them per generated token is one hidden-state vector, about 16KB for a 70B-class model 8,192 values at two bytes each . Even a 512-token prompt moves only about 8MB per boundary. A PCIe 3.0 x1 riser, the kind sold for mining rigs, carries that without noticing. This is why people run six mismatched cards off one motherboard with llama.cpp and it works: layer split tolerates almost any interconnect and any mix of card sizes set the proportions with --tensor-split , for example -ts 3,1 . Tensor parallel: both cards on every layer Tensor parallelism splits each layer instead of each stack of layers. The scheme almost everyone uses comes from Megatron-LM Shoeybi et al., 2019, arXiv:1909.08053 https://arxiv.org/abs/1909.08053?ref=vettedconsumer.com : the feed-forward block's two matrix multiplications are split first by columns and then by rows, and attention is split by heads, so each GPU does its share of every layer with its share of the weights. The cost, in the paper's words, is "only two all-reduces in the forward path" per transformer layer, one after attention and one after the feed-forward block, where every card combines its partial result with the others before anyone can continue. The payoff is that both cards read their half of the weights at the same time, so the bandwidth ceiling doubles. The cost is the count of those synchronisations. For an 80-layer 70B model that is 160 all-reduces per generated token. Each one is small the same 16KB vector , so over one token they move roughly 2.6MB, which is nothing to a PCIe link. What adds up is the round trip: 160 times per token, every GPU stops, exchanges, and waits for the slowest participant. The vLLM parallelism guide https://docs.vllm.ai/en/latest/serving/parallelism scaling.html?ref=vettedconsumer.com draws the conclusion for servers: on nodes whose GPUs lack NVLink, it suggests pipeline parallelism instead of tensor parallelism "for higher throughput and lower communication overhead." What changed in llama.cpp this year llama.cpp has had a --split-mode row option for years, and it earned its reputation: as Johannes Gäßler explains in pull request 19378 https://github.com/ggml-org/llama.cpp/pull/19378?ref=vettedconsumer.com , it "required a synchronization after every single operation," so it was only worth using on old, slow cards such as the P40. That pull request, merged on 9 April 2026, added a real tensor-parallel mode, --split-mode tensor , which infers the data splits from the compute graph and synchronises only where the maths requires. The project's own guide now marks row as deprecated and tensor as experimental, and sums up the choice in one line: "Pipeline-parallel maximizes batch throughput; tensor-parallel minimizes latency." The fine print matters for buyers: - NVIDIA only, in practice. The pull request says only the CUDA backend has the optimisations to beat layer split; the author's own AMD combinations ran worse than the layer baseline, and Vulkan is described as having bad performance. - Install NCCL. Without NVIDIA's collective-communications library, llama.cpp warns that "multi GPU performance will be suboptimal." - No quantised KV cache. Tensor mode requires flash attention and an F16, BF16 or F32 cache. If you were relying on a q8 0 cache to fit long context see the KV cache explainer https://vettedconsumer.com/the-kv-cache-explained-why-long-context-eats-your-vram-and-how-to-fit-more/ , you lose that headroom. - Not every architecture. The guide lists architectures where tensor mode is not implemented, including DeepSeek2, GLM-DSA, MiniMax-M2, Mistral4 and the Mamba hybrids. Several of 2026's big MoE families are on that list, so check before buying cards for them. - No auto-fit. You set the context size yourself. What the measurements say The clearest public numbers come from ikawrakow, who maintains the ik llama.cpp fork and compared mainline's tensor mode against the fork's own "graph" parallel mode in discussion 1247 https://github.com/ikawrakow/ik llama.cpp/discussions/1247?ref=vettedconsumer.com : | Setup | Model | Mainline -sm tensor | ik llama.cpp graph mode | |---|---|---|---| | 4× RTX 3090 | Llama 3 70B, Q4 0, empty context | about 48 tok/s | about 50 tok/s | | 4× RTX 3090 | gpt-oss-120B, MXFP4, empty context | about 123 tok/s | about 164 tok/s | | 4× RTX 3090 | gpt-oss-120B, MXFP4, 57K context | about 99 tok/s | about 126 tok/s | Put the 70B row against the arithmetic. Q4 0 is about 4.5 bits per weight, so roughly 39GB split four ways, and four 3090s reading in parallel set a ceiling near 95 tokens per second. Measured: 48. Tensor parallelism is delivering about half of its theoretical ceiling, and the other half is the synchronisation tax. That is still more than double the dual-card layer-split figure, because the cards read concurrently instead of in turn. For a latency-sensitive single user, this is the first time llama.cpp has offered that trade. The interconnect question has one good public test. Himesh P. ran vLLM's serving benchmark on 3090s https://himeshp.blogspot.com/2025/03/vllm-performance-benchmarks-4x-rtx-3090.html?ref=vettedconsumer.com with and without NVLink bridges, at a 220W power limit, on Qwen2.5-7B: | GPUs | NVLink | Output tok/s | Total throughput tok/s | |---|---|---|---| | 2 | Yes | 715 | 6,790 | | 2 | No | 483 | 4,583 | | 4 | Yes | 535 | 5,093 | | 4 | No | 490 | 4,669 | Two readings. First, for tensor-parallel serving, the NVLink bridge was worth about 48% on a pair. Second, four cards were slower than two for a model this small, and the NVLink advantage shrank to about 9%, because 3090 bridges only connect pairs and the pairs still talk to each other over PCIe. More cards is not automatically more speed when the model already fits on fewer. PCIe lanes, risers and NVLink: when they matter | You run | x4 slot or x1 riser | x8/x16 slots | NVLink bridge 3090 only | |---|---|---|---| | Layer split, one user | Fine for generation; slower model loading | No measurable gain for generation | No benefit | | Tensor parallel, one user | Adds latency to every all-reduce; avoid | Recommended | Helps on a pair | | vLLM serving, many users | Avoid | Recommended | About +48% on a 3090 pair in the test above | Three practical warnings from builders. Flexible PCIe 5.0 risers can drop the link to Gen 4 or Gen 3, a point raised under AZisk's quad RTX Pro 6000 build https://vettedconsumer.com/azisks-quad-rtx-pro-6000-build-384gb-of-vram-and-71-of-every-agent-turn-never-touches-it/ ; it barely matters for layer split and matters a lot for tensor mode. The same build log from Level1Techs needed ACS and IOMMU disabled before NCCL would run tensor parallel. And llama.cpp's guide warns that direct peer-to-peer GPU transfers, which help tensor mode, need driver support "usually restricted to workstation/datacenter GPUs" and "may cause crashes or corrupted outputs" on some motherboards with IOMMU enabled. Budget time for a BIOS session. What to buy - You want to load a model that does not fit on one card, and you chat alone. Any second card with enough memory works, on any slot, in layer mode. Expect single-card speed. A mismatched pair, such as a 3090 plus a 3060, is fine; see what three 3060s measured https://vettedconsumer.com/three-rtx-3060s-vs-one-rtx-3090-for-local-ai-what-a-1-500-build-actually-measured/ . - You want a 70B dense model to feel fast for one user. Buy matched NVIDIA cards, put them in x8 or x16 slots, build llama.cpp with NCCL, and use -sm tensor . Check your model's architecture is supported first. A used 3090 pair with a bridge is still the cheapest way into this 3090 prices here https://vettedconsumer.com/used-rtx-3090-2026-local-ai-best-deal/ . - You serve several users or agents. vLLM with tensor parallel on matched cards; NVLink pays on a pair. This is where multi-GPU scales best, as the serving guide https://vettedconsumer.com/serving-a-local-llm-as-an-api-from-ollamas-endpoint-to-vllm-throughput-and-when-to-rent-instead/ explains. - You run big MoE models. They read only their active parameters per token MoE explained https://vettedconsumer.com/mixture-of-experts-moe-explained-why-active-parameters-decide-what-runs-on-your-machine/ , so layer split is already fast for them, and several MoE families cannot use llama.cpp's tensor mode yet. Capacity matters more than interconnect here. Limits of this synthesis The tensor-mode numbers are from one maintainer's machines with one fork's comparison, and mainline tensor mode is labelled experimental by its own author. The NVLink test used a 7B model under a serving benchmark, not single-user chat. Owner reports for layer split vary with context, driver and build. Treat the ratios as the finding: layer split adds memory, tensor parallel adds speed minus a synchronisation tax, and batching hides that tax. Sources and how we researched this Research: Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism https://arxiv.org/abs/1909.08053?ref=vettedconsumer.com 2019 , section 3 for the column/row split and the two all-reduces per layer; Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism https://arxiv.org/abs/1811.06965?ref=vettedconsumer.com 2018 . Primary project documentation: llama.cpp's multi-GPU guide https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md?ref=vettedconsumer.com , llama.cpp pull request 19378 https://github.com/ggml-org/llama.cpp/pull/19378?ref=vettedconsumer.com , and vLLM's parallelism and scaling guide https://docs.vllm.ai/en/latest/serving/parallelism scaling.html?ref=vettedconsumer.com . Measurements: ik llama.cpp discussion 1247 https://github.com/ikawrakow/ik llama.cpp/discussions/1247?ref=vettedconsumer.com , Himesh P.'s vLLM 3090 benchmarks https://himeshp.blogspot.com/2025/03/vllm-performance-benchmarks-4x-rtx-3090.html?ref=vettedconsumer.com , and the r/LocalLLaMA dual-3090 70B report https://www.reddit.com/r/LocalLLaMA/comments/1he2v2n/?ref=vettedconsumer.com . Ceilings are this site's bandwidth arithmetic memory bandwidth divided by bytes read per token , not measurements. We have not tested these configurations. Non-commercial site, no retailer links. Related: Prompt processing vs generation https://vettedconsumer.com/prompt-processing-vs-generation-why-your-box-is-fast-at-one-and-slow-at-the-other/ · Why is my local LLM slow https://vettedconsumer.com/why-is-my-local-llm-slow-the-6-bottlenecks-in-order/ · Intel Arc Pro B60: 192GB the cheap way https://vettedconsumer.com/intel-arc-pro-b60-192gb-of-vram-the-cheap-way-and-what-it-really-costs/ · RTX 5090 vs RTX 4090 https://vettedconsumer.com/rtx-5090-vs-rtx-4090-for-local-llms-what-32gb-and-78-more-bandwidth-really-buy/