DP, TP, PP and EP: How LLM Parallelism Actually Fits Together A GPU trace of a Mixture-of-Experts model served on 8 GPUs showed every GPU running all 256 experts as thin 1/8 slices rather than distributing experts across devices, an analysis of Qwen3.5-35B-A3B explains. In that model, roughly 32.2 billion of the 35 billion parameters are experts (256 experts × 3.15 million params × 40 layers), while each token touches only 8 of 256 experts per layer, so memory is dominated by experts and per-token compute stays small. The breakdown compares data, tensor, pipeline and expert parallelism by what each communicates and how often, with tensor and expert parallelism on the critical path every layer while data parallelism's gradient all-reduce runs once per training step. I was looking at a GPU trace of a Mixture-of-Experts model served on 8 GPUs, and one thing didn’t make sense to me. The model has 256 experts. I had always pictured MoE serving with the experts spread out, a few dozen on each GPU, and tokens travelling to wherever their experts live. That’s not what the trace showed. Every GPU was running all 256 experts, and each one was a thin 1/8 slice of the real expert. The matrix multiplications were narrow, and there were a lot of them. That raised a set of questions I couldn’t answer: This post is what I found. I’ll explain each kind of parallelism, when you need it and how they combine, then work through one real model on 8 GPUs with actual numbers. Qwen3.5–35B-A3B : about 35B parameters in total, about 3B active per token. Layers 40 30 linear-attention "GDN" layers + 10 full-attention layers Attention 16 query heads, 2 KV heads, head dim 256MoE every layer: 256 experts, each token uses the top 8Hidden size H = 2048Expert MLP SwiGLU, intermediate size I = 512 gate up: 2048 x 1024 2.10 M params down: 512 x 2048 1.05 M params ≈ 3.15 M params per expert One number explains most of what follows: 256 experts x 3.15 M params x 40 layers ≈ 32.2 B params So about 32 of the 35 billion parameters are experts , yet each token touches only 8 of the 256 experts per layer. Memory is dominated by experts, while compute per token is small. Keep that in mind. Each form of parallelism answers one question: which part of the work do we split across GPUs? ┌── the batch ──────────► Data Parallel DP the work of one step ───┼── the weight matrices ► Tensor Parallel TP ├── the layers ─────────► Pipeline Parallel PP └── the experts ────────► Expert Parallel EP, MoE only There are also a few variants you’ll see in configs: The four core strategies differ mainly in what they communicate, and how often : Strategy Collective How often On the critical path?-------- ---------------------------- -------------------------- ---------------------DP all-reduce of gradients once per training step no overlapped TP all-reduce of activations every layer YESPP point-to-point send/recv once per stage boundary partly idle "bubble" EP all-to-all of routed tokens twice per MoE layer YES Almost every layout decision follows from this table. Each GPU holds a full copy of the model and processes different samples. global batch ──┬──► GPU 0: full model ──► gradients₀ ──┐ ├──► GPU 1: full model ──► gradients₁ ──┼──► all-reduce average ├──► GPU 2: full model ──► gradients₂ ──┤ │ └──► GPU 3: full model ──► gradients₃ ──┘ ▼ every GPU applies the same update Training memory with mixed-precision Adam is about 16 bytes per parameter: 2 for bf16 weights, 2 for bf16 gradients and 12 for the fp32 master copy plus Adam’s two moments. ZeRO shards these across the DP group: Plain DP 16 B/param on every GPUZeRO-1 4 + 12/DP shard optimizer states ZeRO-2 2 + 14/DP also shard gradients ZeRO-3 16/DP also shard weights; all-gather them layer by layer TP splits each layer’s weight matrices across GPUs so they compute every layer together. The standard trick from Megatron-LM pairs a column split with a row split so that only one sync is needed per block: MLP: Y = act X · A · B with TP = 2 A split by COLUMNS: A = A₀ | A₁ GPU r computes Zᵣ = act X · Aᵣ ← no communicationB split by ROWS: B = B₀ ; B₁ GPU r computes Yᵣ = Zᵣ · Bᵣ Y = Y₀ + Y₁ ← one all-reduce Attention works the same way. Q/K/V are split by heads , so each GPU owns a subset of heads, and the output projection is row-split and followed by one all-reduce. For Qwen3.5–35B-A3B with TP = 2, each GPU owns 8 of the 16 query heads and 1 of the 2 KV heads. Cost. Every all-reduce moves the full activation, T tokens x 2048 x 2 bytes, which is 4 KB per token per all-reduce . It happens every layer, and nothing can proceed until it finishes. That is why TP needs very fast GPU-to-GPU links NVLink, xGMI and similar and almost always stays inside one server . The KV-head limit. This one surprised me. TP splits attention by KV heads, so it can’t usefully go beyond the number of KV heads: Qwen3.5-35B-A3B has 2 KV heads.TP = 2 → GPU 0 gets KV head 0, GPU 1 gets KV head 1 clean splitTP = 8 → 8 GPUs, only 2 KV heads → each head copied to 4 GPUs → the KV cache is stored 4 times Keep this in mind. It’s the main reason the 8-GPU example below doesn’t simply use TP = 8. Sequence Parallelism SP comes paired with TP. Outside the matmuls LayerNorm, residual adds , activations are split along the sequence instead of being copied. The all-reduce becomes a reduce-scatter plus an all-gather. The traffic is the same, but activation memory per GPU drops by the TP factor. Each GPU or group of GPUs owns a contiguous block of layers, called a stage . Activations flow forward through the stages and gradients flow back. stage 0: layers 0-19 ──activations──► stage 1: layers 20-39 ◄──gradients──── The problem is the bubble . With a single batch, stage 1 waits while stage 0 works, and the reverse happens on the way back. The fix is to cut the batch into many micro-batches and interleave them: 1F1B schedule, 2 stages, 4 micro-batches F = forward, B = backward, .. = idle stage 0: F1 F2 F3 F4 .. B1 B2 B3 B4 stage 1: .. F1 B1 F2 B2 F3 B3 F4 B4 ^^ ^^ bubble bubble fraction = p - 1 / m + p - 1 p = stages, m = micro-batchesp = 2, m = 16 → 1/17 ≈ 6 % In a MoE layer, the dense MLP is replaced by many expert MLPs plus a small router . The router picks the top-k experts for each token, and the token’s output is a weighted sum of what those experts produce: y = Σ gᵢ x · Expertᵢ x over the top-8 experts chosen for token x EP gives each GPU a subset of whole experts . With 256 experts and EP = 8, that’s 32 experts per GPU. Tokens are then sent to wherever their experts live: GPU 0 GPU 1 GPU 7 its tokens its tokens ... its tokens │ │ │ router router router top-8 of 256 per token │ │ │ ════════════════ all-to-all DISPATCH ══════════════════════ token → GPU expert id ÷ 32 │ │ │ experts 0-31 experts 32-63 ... experts 224-255 one grouped GEMM per GPU │ │ │ ════════════════ all-to-all COMBINE ══════════════════════ results go back home │ │ │ weighted sum weighted sum weighted sum Cost. There are two all-to-alls per MoE layer. A token is 4 KB 2048 values in bf16 . Good kernels for example DeepEP send one copy per destination GPU , not per expert. With 8 experts out of 256 spread across 8 GPUs, a token lands on about 5.3 distinct GPUs on average: distinct GPUs per token = EP · 1 - C E - E/EP, k / C E, k = 8 · 1 - C 224, 8 / C 256, 8 ≈ 8 · 1 - 0.338 ≈ 5.3 The weakness is load imbalance. If the router sends a lot of tokens to a few popular experts, the GPUs holding them become stragglers, and everyone waits at the combine step. The usual fixes: EP versus TP on the experts. This was my original question. Without EP, frameworks apply TP to the MoE layers too: every GPU holds all experts, each one sliced. That’s exactly what my trace showed. TP = 8 on experts EP = 8 ------------------ ------------------experts per GPU 256 each 1/8 slice 32 whole gate up GEMM per expert 2048 x 128 2048 x 1024down GEMM per expert 64 x 2048 512 x 2048expert weight bytes per GPU 1/8 of total 1/8 of total ← same collective per MoE layer all-reduce, all tokens 2 x all-to-all, routed tokens onlyload balance perfect depends on router For a fixed batch, the expert weight bytes per GPU are the same , and so is the average number of rows each expert processes. What EP changes is: A practical way to decide, in order: 1. Does the model plus optimizer / KV cache fit on one GPU? YES → use DP only. Done. training: add ZeRO if optimizer states don't fit 2. Is it a MoE model? YES → use EP for the expert weights. Experts dominate memory, and EP divides them. Keep the EP group inside one server if possible all-to-all is latency-critical .3. Attention / dense layers still too big, or need lower latency? → use TP inside one server, up to the number of KV heads. → beyond that, use DP-attention instead of more TP avoids copying the KV cache .4. Still doesn't fit, or the job spans several servers? → add PP across servers cheap on bandwidth . Use many micro-batches to keep the bubble small.5. Very long sequences 100K+ tokens ? → add CP to split the sequence for attention.6. Fill whatever GPUs remain with DP and, for experts, expert-DP . The rule underneath all of this: Put the collectives that run every layer TP, EP on the fastest links. Let the ones that run rarely DP, PP cross the slower network. Some symptoms and their usual fixes: Symptom Usual fix------------------------------------------- ------------------------------------------------TP all-reduce dominates the timeline lower TP; keep it in-node; switch attention to DPAll-to-all dominates prefill overlap comm with compute; keep EP in-nodeAll-to-all dominates decode low-latency dispatch kernels, CUDA graphs, bigger batchOne GPU is always the slowest in MoE layers expert imbalance → EPLB / redundant expertsGPUs idle at the start/end of each step pipeline bubble → more micro-batches, interleavingOut of KV-cache memory with TP KV heads DP-attention For dense layers , the GPUs form a grid: total GPUs = TP x PP x DP For MoE layers , here is the part that took me longest to see. EP is not an extra dimension of the grid. It reuses the same GPUs as TP and DP inside each pipeline stage and regroups them for the expert weights: TP x DP = ETP x EP x EDPETP = tensor parallelism inside each expert usually 1 for small experts EP = how many GPUs the experts are spread overEDP = expert data parallelism: how many copies of each expert shard exist expert gradients are averaged only across these copies So a single GPU belongs to several groups at once: a TP group and a DP group for attention, an EP group and an EDP group for experts, and one pipeline stage. Frameworks number GPUs in a fixed nesting order. Megatron-LM uses tp → cp → ep → dp → pp, with TP varying fastest. GPUs with neighbouring rank numbers sit in the same server, so TP and EP naturally stay inside a server, while DP and PP span servers. That's exactly what we want. Before the 8-GPU serving example, here is what a layout with all four strategies looks like when training this model: PP = 2 one pipeline stage per 2 servers, 20 layers each 15 GDN + 5 full-attention TP = 2 matches the 2 KV heads exactlyDP = 8 32 / 2 x 2 EP = 8 one server = one EP group → 32 experts per GPUEDP = 2 because TP x DP = 16 = ETP 1 x EP 8 x EDP 2 PP stage 0 layers 0-19 PP stage 1 layers 20-39 ┌───────── Node 0 ─────────┬───────── Node 1 ──────────┐ ┌── Node 2 ──┬── Node 3 ──┐rank: 0 1 2 3 4 5 6 7 │ 8 9 10 11 12 13 14 15 16 ...... 23 24 ...... 31TP=2: 0 1 2 3 4 5 6 7 │ 8 9 10 11 12 13 14 15 same pattern DP=8: one TP pair = one DP replica → pairs {0,1},{2,3},...,{14,15} dense DP groups: {0,2,4,...,14} and {1,3,5,...,15}EP=8: group A = {0..7} │ group B = {8..15} {16..23} {24..31} r0: E0-31 r1: E32-63 r2: E64-95 ... r7: E224-255 r8..r15 use the same split EDP=2: 0,8 1,9 2,10 ... 7,15 ← ranks holding the same experts, on different nodesPP p2p: rank i ⇄ rank i+16 ← stage boundary, inter-node One training step on this layout: DP: global batch split 8 ways → each TP pair gets its own micro-batches │ ▼ stage 0, for each of its 20 layers: │ attention / GDN on the TP pair → reduce-scatter 2 GPUs, in-server │ router picks top-8 experts per token │ all-to-all DISPATCH → 8 GPUs in-server │ grouped GEMM on 32 local experts │ all-to-all COMBINE → 8 GPUs in-server │ all-gather for the next attention → 2 GPUs in-server │ ▼ PP: send activations, rank i → rank i+16 cross-server │ stage 1 runs layers 20-39 → loss │ ▼ backward: same collectives in reverse 1F1B, m = 16 → bubble ≈ 6 % │ ▼ gradient sync, overlapped with backward: cross-server attention / dense grads → all-reduce across DP = 8 expert grads → all-reduce across EDP = 2 The collectives that run every layer TP and EP never leave a server. Only the pipeline hand-off and the once-per-step gradient sync cross the network. What EP buys here: per GPU, experts: 32 experts x 20 layers x 3.15 M ≈ 2.0 B params ≈ 4 GB bf16 without EP: 256 experts x 20 layers x 3.15 M ≈ 16 B params ≈ 32 GB bf16 Now the case that started all of this: serving the model on one 8-GPU server. 35 B params x 2 bytes bf16 ≈ 70 GB of weights That fits comfortably across 8 GPUs, so no PP . Everything stays inside one server, which is great for latency. The simplest launch is TP = 8 for everything. This is what my original trace was: Attention: TP = 8 → 16 q heads / 8 = 2 per GPU 2 KV heads / 8 → not possible → each KV head copied to 4 GPUsMoE: TP = 8 → every GPU holds all 256 experts, each sliced to 1/8 gate up per expert: 2048 x 128, down: 64 x 2048 Two problems: Attention: DP = 4 x TP = 2 → 4 independent request groups; TP = 2 matches the 2 KV headsMoE: EP = 8 → each GPU holds 32 whole expertsPP: 1 The rank layout: One server, 8 GPUs PP = 1 ┌──────────────────────────────────────────────────────────────────┐rank: 0 1 2 3 4 5 6 7ATTENTION DP = 4 x TP = 2 : 0 1 2 3 4 5 6 7 DP grp 0 DP grp 1 DP grp 2 DP grp 3 ← different requests in each r0: kv head 0 + q heads 0-7, r1: kv head 1 + q heads 8-15 same in every pair MoE EP = 8, one group spanning the whole server : r0: E0-31 r1: E32-63 r2: E64-95 r3: E96-127 r4: E128-159 r5: E160-191 r6: E192-223 r7: E224-255Same GPU, two roles: rank 5 = "TP rank 1 of DP group 2" for attention = "owner of experts 160-191" for MoE └──────────────────────────────────────────────────────────────────┘ KV cache per token, across the 10 full-attention layers: 2 K and V x 2 KV heads x 256 dim x 2 bytes x 10 layers = 20 KB per token KV stored per token all 8 GPUs duplicationTP = 8 80 KB 4x each KV head on 4 GPUs DP = 4 x TP = 2 20 KB 1x each request lives in one pair That’s about 4x more tokens in the KV cache , which means more concurrent requests. Weight memory per GPU: TP = 8 DP-attn + EP = 8experts 32.2 B / 8 ≈ 4.0 B 32.2 B / 8 ≈ 4.0 B ≈ 8 GBattention / other ≈ 2.8 B / 8 ≈ 0.35 B ≈ 2.8 B / 2 ≈ 1.4 B copied in each DP group total weights ≈ 8.7 GB ≈ 10.8 GB The EP layout spends about 2 GB more per GPU on weights, because the attention weights are copied across the 4 DP groups. In exchange it saves 4x on the KV cache, which is usually the much larger number. The 30 GDN layers also keep a fixed-size recurrent state per sequence. It’s split differently in the two layouts, but it’s similar in total. Suppose 64 requests are running, 16 in each DP group. Each request produces one token per step: each DP group: its own batch of B g tokens 1 per running request B g = 16, B total = 64 │ ▼ attention / GDN in the TP pair → all-reduce 2 GPUs │ each pair works only on its own 16 requests' KV cache │ ▼ router: top-8 per token │ 16 tokens x 8 = 128 token, expert pairs per group │ ▼ all-to-all DISPATCH across all 8 GPUs tokens from all 4 DP groups mix here │ each GPU sends ≈ 8 tokens x 5.3 GPUs x 4 KB ≈ 170 KB → tiny, latency-bound │ ▼ each GPU: grouped GEMM on its 32 experts │ rows per expert ≈ Σ B g · 8 / 256 = B total / 32 = 2 │ experts actually hit per step ≈ 256 · 1 - 1 - 1/32 ^64 ≈ 222 of 256 │ ▼ all-to-all COMBINE → tokens return to their DP group │ ▼ weighted sum of 8 expert outputs + residual → next layer x 40 layers → 80 all-to-alls per generated token What decode tells us: Now suppose each DP group is prefilling 4 prompts of 1,024 tokens, which is 4,096 tokens per group and 16,384 in total: each DP group: 4 prompts x 1,024 tokens = 4,096 tokens │ ▼ attention / GDN in the TP pair → all-reduce 2 GPUs │ 4,096 tokens x 4 KB = 16 MB per all-reduce → bandwidth matters now │ ▼ router: top-8 per token │ ▼ all-to-all DISPATCH across all 8 GPUs │ each GPU sends ≈ 2,048 tokens x 5.3 GPUs x 4 KB ≈ 42 MB per layer │ ▼ each GPU: grouped GEMM on its 32 experts │ rows per expert ≈ 16,384 x 8 / 256 = 512 │ gate up: 512 x 2048 · 2048 x 1024 ← a real, compute-bound GEMM │ down: 512 x 512 · 512 x 2048 │ ▼ all-to-all COMBINE → same ≈ 42 MB back What prefill tells us: TP = 8 DP-attn 4x2 + EP = 8---------------------------- ------------------------- ---------------------------attention split 2 q heads, KV copied 4x 8 q heads, 1 KV head, no copyKV cache per token total 80 KB 20 KBexperts per GPU 256 slices 1/8 each 32 whole expertsexpert GEMM shape gate up 2048 x 128 2048 x 1024expert weight bytes per GPU same samerows per expert same batch same sameMoE collective all-reduce of all tokens 2 x all-to-all, routed onlyload balance perfect router-dependentweights per GPU ≈ 8.7 GB ≈ 10.8 GBmain win simplicity, balance 4x KV room → bigger batches, better expert GEMM shapes If you need more throughput, there are two directions: PP only enters the picture if a single replica can’t fit in one server, which isn’t the case for this model. What I originally saw in the trace every GPU running thin slices of all 256 experts wasn’t wrong. It was simply TP = 8 applied everywhere. Once you see the four strategies as four different ways of splitting the work, the better layout for this model is clear. DP, TP, PP and EP: How LLM Parallelism Actually Fits Together https://pub.towardsai.net/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together-2cc4e611da72 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.