Does a Second GPU Make Your Local LLM Faster? Layer Split vs Tensor Parallel, Explained

Editor checklist: every measured number below is attributed and linked; the ceilings are arithmetic from published memory bandwidth, labelled as such. We have not run these configurations. Delete this box before publishing.

The most common question from people pricing a second RTX 3090 is whether it will make their model faster. The answer depends on a setting most of them have never touched. llama.cpp can split a model across GPUs in two fundamentally different ways, and on a single chat stream one of them adds memory and almost no speed, while the other can roughly double throughput at the cost of constant chatter between the cards. Which one you get decides whether PCIe lanes, risers and NVLink bridges matter at all. This is how both work, what the measurements say, and what to buy.

The short answer #

• Layer split (llama.cpp's default) gives you the combined memory of both cards. For one user generating one reply, it runs at roughly the speed of a single card, because the cards take turns.
• Tensor parallel makes both cards work on every token at once. It can nearly double single-stream speed on two cards, but every layer ends in a synchronisation, so the link between the cards starts to matter.
• Serving many users changes the maths again: batching hides the synchronisation cost, and multi-GPU throughput scales well either way.

Layer split: two cards taking turns #

In llama.cpp's default mode, --split-mode layer, each GPU holds a contiguous block of the model's layers along with the KV cache for those layers. The project's multi-GPU guide calls this pipeline parallelism and describes its trade plainly: it "minimizes data transfers between GPUs but requires many tokens to scale well." The idea goes back to GPipe (Huang et al., 2018, arXiv:1811.06965), which placed different sub-sequences of layers on separate accelerators and kept them all busy by feeding the pipeline many micro-batches at once.

Generation is the case where there are no micro-batches. To produce one token, card one runs its 40 layers, hands a single activation vector to card two, and waits. Card two runs its 40 layers, and the token comes out. At any instant one card is working and the other is idle. Since decode speed on a GPU is set by how fast it can read its weights (the bandwidth-not-TFLOPS rule), two cards reading half the model each, one after the other, take the same time as one imaginary card with twice the memory reading all of it.

The numbers line up. A Llama 70B at Q4_K_M is about 42GB of weights (70 billion × 4.8 bits ÷ 8). A 3090 reads memory at 936GB/s, so the ceiling for layer-split generation across two of them is 936 ÷ 42, about 22 tokens per second. Owners on r/LocalLLaMA report 17.9 tokens per second at short context on a dual-3090 box, falling to 7.9 at 32K. That is a healthy fraction of the ceiling, and it is also exactly what one 48GB card with 3090 bandwidth would do. The second card bought the capacity to load the model, not speed. Our earlier two 3090s versus one 5090 comparison is really this effect in buying-guide form.

Layer split has a large practical upside: the cards barely talk. What crosses between them per generated token is one hidden-state vector, about 16KB for a 70B-class model (8,192 values at two bytes each). Even a 512-token prompt moves only about 8MB per boundary. A PCIe 3.0 x1 riser, the kind sold for mining rigs, carries that without noticing. This is why people run six mismatched cards off one motherboard with llama.cpp and it works: layer split tolerates almost any interconnect and any mix of card sizes (set the proportions with --tensor-split, for example -ts 3,1).

Tensor parallel: both cards on every layer #

Tensor parallelism splits each layer instead of each stack of layers. The scheme almost everyone uses comes from Megatron-LM (Shoeybi et al., 2019, arXiv:1909.08053): the feed-forward block's two matrix multiplications are split first by columns and then by rows, and attention is split by heads, so each GPU does its share of every layer with its share of the weights. The cost, in the paper's words, is "only two all-reduces in the forward path" per transformer layer, one after attention and one after the feed-forward block, where every card combines its partial result with the others before anyone can continue.

The payoff is that both cards read their half of the weights at the same time, so the bandwidth ceiling doubles. The cost is the count of those synchronisations. For an 80-layer 70B model that is 160 all-reduces per generated token. Each one is small (the same 16KB vector), so over one token they move roughly 2.6MB, which is nothing to a PCIe link. What adds up is the round trip: 160 times per token, every GPU stops, exchanges, and waits for the slowest participant. The vLLM parallelism guide draws the conclusion for servers: on nodes whose GPUs lack NVLink, it suggests pipeline parallelism instead of tensor parallelism "for higher throughput and lower communication overhead."

What changed in llama.cpp this year #

llama.cpp has had a --split-mode row option for years, and it earned its reputation: as Johannes Gäßler explains in pull request 19378, it "required a synchronization after every single operation," so it was only worth using on old, slow cards such as the P40. That pull request, merged on 9 April 2026, added a real tensor-parallel mode, --split-mode tensor, which infers the data splits from the compute graph and synchronises only where the maths requires. The project's own guide now marks row as deprecated and tensor as experimental, and sums up the choice in one line: "Pipeline-parallel maximizes batch throughput; tensor-parallel minimizes latency."

The fine print matters for buyers:

• NVIDIA only, in practice. The pull request says only the CUDA backend has the optimisations to beat layer split; the author's own AMD combinations ran worse than the layer baseline, and Vulkan is described as having bad performance.
• Install NCCL. Without NVIDIA's collective-communications library, llama.cpp warns that "multi GPU performance will be suboptimal."
• No quantised KV cache. Tensor mode requires flash attention and an F16, BF16 or F32 cache. If you were relying on a q8_0 cache to fit long context (seethe KV cache explainer ), you lose that headroom.
• Not every architecture. The guide lists architectures where tensor mode is not implemented, including DeepSeek2, GLM-DSA, MiniMax-M2, Mistral4 and the Mamba hybrids. Several of 2026's big MoE families are on that list, so check before buying cards for them.
• No auto-fit. You set the context size yourself.

What the measurements say #

The clearest public numbers come from ikawrakow, who maintains the ik_llama.cpp fork and compared mainline's tensor mode against the fork's own "graph" parallel mode in discussion 1247:

Put the 70B row against the arithmetic. Q4_0 is about 4.5 bits per weight, so roughly 39GB split four ways, and four 3090s reading in parallel set a ceiling near 95 tokens per second. Measured: 48. Tensor parallelism is delivering about half of its theoretical ceiling, and the other half is the synchronisation tax. That is still more than double the dual-card layer-split figure, because the cards read concurrently instead of in turn. For a latency-sensitive single user, this is the first time llama.cpp has offered that trade.

The interconnect question has one good public test. Himesh P. ran vLLM's serving benchmark on 3090s with and without NVLink bridges, at a 220W power limit, on Qwen2.5-7B:

Two readings. First, for tensor-parallel serving, the NVLink bridge was worth about 48% on a pair. Second, four cards were slower than two for a model this small, and the NVLink advantage shrank to about 9%, because 3090 bridges only connect pairs and the pairs still talk to each other over PCIe. More cards is not automatically more speed when the model already fits on fewer.

PCIe lanes, risers and NVLink: when they matter #

Three practical warnings from builders. Flexible PCIe 5.0 risers can drop the link to Gen 4 or Gen 3, a point raised under AZisk's quad RTX Pro 6000 build; it barely matters for layer split and matters a lot for tensor mode. The same build log from Level1Techs needed ACS and IOMMU disabled before NCCL would run tensor parallel. And llama.cpp's guide warns that direct peer-to-peer GPU transfers, which help tensor mode, need driver support "usually restricted to workstation/datacenter GPUs" and "may cause crashes or corrupted outputs" on some motherboards with IOMMU enabled. Budget time for a BIOS session.

What to buy #

• You want to load a model that does not fit on one card, and you chat alone. Any second card with enough memory works, on any slot, in layer mode. Expect single-card speed. A mismatched pair, such as a 3090 plus a 3060, is fine; seewhat three 3060s measured .
• You want a 70B dense model to feel fast for one user. Buy matched NVIDIA cards, put them in x8 or x16 slots, build llama.cpp with NCCL, and use-sm tensor . Check your model's architecture is supported first. A used 3090 pair with a bridge is still the cheapest way into this (3090 prices here ).
• You serve several users or agents. vLLM with tensor parallel on matched cards; NVLink pays on a pair. This is where multi-GPU scales best, as theserving guide explains.
• You run big MoE models. They read only their active parameters per token (MoE explained ), so layer split is already fast for them, and several MoE families cannot use llama.cpp's tensor mode yet. Capacity matters more than interconnect here.

Limits of this synthesis #

The tensor-mode numbers are from one maintainer's machines with one fork's comparison, and mainline tensor mode is labelled experimental by its own author. The NVLink test used a 7B model under a serving benchmark, not single-user chat. Owner reports for layer split vary with context, driver and build. Treat the ratios as the finding: layer split adds memory, tensor parallel adds speed minus a synchronisation tax, and batching hides that tax.

Sources and how we researched this #

Research: Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019), section 3 for the column/row split and the two all-reduces per layer; Huang et al., GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism (2018). Primary project documentation: llama.cpp's multi-GPU guide, llama.cpp pull request 19378, and vLLM's parallelism and scaling guide. Measurements: ik_llama.cpp discussion 1247, Himesh P.'s vLLM 3090 benchmarks, and the r/LocalLLaMA dual-3090 70B report. Ceilings are this site's bandwidth arithmetic (memory bandwidth divided by bytes read per token), not measurements. We have not tested these configurations. Non-commercial site, no retailer links.

Related: Prompt processing vs generation · Why is my local LLM slow · Intel Arc Pro B60: 192GB the cheap way · RTX 5090 vs RTX 4090