I was looking at a GPU trace of a Mixture-of-Experts model served on 8 GPUs, and one thing didnβt make sense to me.
The model has 256 experts. I had always pictured MoE serving with the experts spread out, a few dozen on each GPU, and tokens travelling to wherever their experts live. Thatβs not what the trace showed. Every GPU was running all 256 experts, and each one was a thin 1/8 slice of the real expert. The matrix multiplications were narrow, and there were a lot ofΒ them.
That raised a set of questions I couldnβt answer:
This post is what I found. Iβll explain each kind of parallelism, when you need it and how they combine, then work through one real model on 8 GPUs with actual numbers.
Qwen3.5β35B-A3B: about 35B parameters in total, about 3B active perΒ token.
Layers 40 (30 linear-attention "GDN" layers + 10 full-attention layers)Attention 16 query heads, 2 KV heads, head dim 256MoE every layer: 256 experts, each token uses the top 8Hidden size H = 2048Expert MLP SwiGLU, intermediate size I = 512 gate_up: 2048 x 1024 (2.10 M params) down: 512 x 2048 (1.05 M params) β 3.15 M params per expert
One number explains most of what follows:
256 experts x 3.15 M params x 40 layers β 32.2 B params
So about 32 of the 35 billion parameters are experts, yet each token touches only 8 of the 256 experts per layer. Memory is dominated by experts, while compute per token is small. Keep that inΒ mind.
Each form of parallelism answers one question: which part of the work do we split acrossΒ GPUs?
βββ the batch βββββββββββΊ Data Parallel (DP) the work of one step ββββΌββ the weight matrices βΊ Tensor Parallel (TP) βββ the layers ββββββββββΊ Pipeline Parallel (PP) βββ the experts βββββββββΊ Expert Parallel (EP, MoE only)
There are also a few variants youβll see in configs:
The four core strategies differ mainly in what they communicate, and howΒ often:
Strategy Collective How often On the critical path?-------- ---------------------------- -------------------------- ---------------------DP all-reduce of gradients once per training step no (overlapped)TP all-reduce of activations every layer YESPP point-to-point send/recv once per stage boundary partly (idle "bubble")EP all-to-all of routed tokens twice per MoE layer YES
Almost every layout decision follows from this table.
Each GPU holds a full copy of the model and processes different samples.
global batch βββ¬βββΊ GPU 0: full model βββΊ gradientsβ βββ ββββΊ GPU 1: full model βββΊ gradientsβ βββΌβββΊ all-reduce (average) ββββΊ GPU 2: full model βββΊ gradientsβ βββ€ β ββββΊ GPU 3: full model βββΊ gradientsβ βββ βΌ every GPU applies the same update
Training memory with mixed-precision Adam is about 16 bytes per parameter: 2 for bf16 weights, 2 for bf16 gradients and 12 for the fp32 master copy plus Adamβs two moments. ZeRO shards these across the DP group:
Plain DP 16 B/param on every GPUZeRO-1 4 + 12/DP (shard optimizer states)ZeRO-2 2 + 14/DP (also shard gradients)ZeRO-3 16/DP (also shard weights; all-gather them layer by layer)
TP splits each layerβs weight matrices across GPUs so they compute every layer together. The standard trick (from Megatron-LM) pairs a column split with a row split so that only one sync is needed perΒ block:
MLP: Y = act(X Β· A) Β· B with TP = 2 A split by COLUMNS: A = [Aβ | Aβ] GPU r computes Zα΅£ = act(X Β· Aα΅£) β no communicationB split by ROWS: B = [Bβ ; Bβ] GPU r computes Yα΅£ = Zα΅£ Β· Bα΅£ Y = Yβ + Yβ β one all-reduce
Attention works the same way. Q/K/V are split by heads, so each GPU owns a subset of heads, and the output projection is row-split and followed by one all-reduce.
For Qwen3.5β35B-A3B with TP = 2, each GPU owns 8 of the 16 query heads and 1 of the 2 KV heads.
Cost. Every all-reduce moves the full activation, T tokens x 2048 x 2 bytes, which is 4 KB per token per all-reduce. It happens every layer, and nothing can proceed until it finishes. That is why TP needs very fast GPU-to-GPU links (NVLink, xGMI and similar) and almost always stays inside oneΒ server.
The KV-head limit. This one surprised me. TP splits attention by KV heads, so it canβt usefully go beyond the number of KVΒ heads:
Qwen3.5-35B-A3B has 2 KV heads.TP = 2 β GPU 0 gets KV head 0, GPU 1 gets KV head 1 clean splitTP = 8 β 8 GPUs, only 2 KV heads β each head copied to 4 GPUs β the KV cache is stored 4 times
Keep this in mind. Itβs the main reason the 8-GPU example below doesnβt simply use TP = 8.
Sequence Parallelism (SP) comes paired with TP. Outside the matmuls (LayerNorm, residual adds), activations are split along the sequence instead of being copied. The all-reduce becomes a reduce-scatter plus an all-gather. The traffic is the same, but activation memory per GPU drops by the TPΒ factor.
Each GPU (or group of GPUs) owns a contiguous block of layers, called a stage. Activations flow forward through the stages and gradients flowΒ back.
stage 0: layers 0-19 ββactivationsβββΊ stage 1: layers 20-39 βββgradientsββββ
The problem is the bubble. With a single batch, stage 1 waits while stage 0 works, and the reverse happens on the way back. The fix is to cut the batch into many micro-batches and interleave them:
1F1B schedule, 2 stages, 4 micro-batches (F = forward, B = backward, .. = idle) stage 0: F1 F2 F3 F4 .. B1 B2 B3 B4 stage 1: .. F1 B1 F2 B2 F3 B3 F4 B4 ^^ ^^ bubble
bubble fraction = (p - 1) / (m + p - 1) p = stages, m = micro-batchesp = 2, m = 16 β 1/17 β 6 %
In a MoE layer, the dense MLP is replaced by many expert MLPs plus a small router. The router picks the top-k experts for each token, and the tokenβs output is a weighted sum of what those expertsΒ produce:
y = Ξ£ gα΅’(x) Β· Expertα΅’(x) over the top-8 experts chosen for token x
EP gives each GPU a subset of whole experts. With 256 experts and EP = 8, thatβs 32 experts per GPU. Tokens are then sent to wherever their expertsΒ live:
GPU 0 GPU 1 GPU 7 its tokens its tokens ... its tokens β β β router router router top-8 of 256 per token β β β ββββββββββββββββ all-to-all DISPATCH ββββββββββββββββββββββ token β GPU (expert_id Γ· 32) β β β experts 0-31 experts 32-63 ... experts 224-255 one grouped GEMM per GPU β β β ββββββββββββββββ all-to-all COMBINE ββββββββββββββββββββββ results go back home β β β weighted sum weighted sum weighted sum
Cost. There are two all-to-alls per MoE layer. A token is 4 KB (2048 values in bf16). Good kernels (for example DeepEP) send one copy per destination GPU, not per expert. With 8 experts out of 256 spread across 8 GPUs, a token lands on about 5.3 distinct GPUs onΒ average:
distinct GPUs per token = EP Β· (1 - C(E - E/EP, k) / C(E, k)) = 8 Β· (1 - C(224, 8) / C(256, 8)) β 8 Β· (1 - 0.338) β 5.3
The weakness is load imbalance. If the router sends a lot of tokens to a few popular experts, the GPUs holding them become stragglers, and everyone waits at the combine step. The usualΒ fixes:
EP versus TP on the experts. This was my original question. Without EP, frameworks apply TP to the MoE layers too: every GPU holds all experts, each one sliced. Thatβs exactly what my traceΒ showed.
TP = 8 on experts EP = 8 ------------------ ------------------experts per GPU 256 (each 1/8 slice) 32 (whole)gate_up GEMM per expert 2048 x 128 2048 x 1024down GEMM per expert 64 x 2048 512 x 2048expert weight bytes per GPU 1/8 of total 1/8 of total β same!collective per MoE layer all-reduce, all tokens 2 x all-to-all, routed tokens onlyload balance perfect depends on router
For a fixed batch, the expert weight bytes per GPU are the same, and so is the average number of rows each expert processes. What EP changesΒ is:
A practical way to decide, in order:
1. Does the model (plus optimizer / KV cache) fit on one GPU? YES β use DP only. Done. (training: add ZeRO if optimizer states don't fit)2. Is it a MoE model? YES β use EP for the expert weights. Experts dominate memory, and EP divides them. Keep the EP group inside one server if possible (all-to-all is latency-critical).3. Attention / dense layers still too big, or need lower latency? β use TP inside one server, up to the number of KV heads. β beyond that, use DP-attention instead of more TP (avoids copying the KV cache).4. Still doesn't fit, or the job spans several servers? β add PP across servers (cheap on bandwidth). Use many micro-batches to keep the bubble small.5. Very long sequences (100K+ tokens)? β add CP to split the sequence for attention.6. Fill whatever GPUs remain with DP (and, for experts, expert-DP).
The rule underneath all of this:
Put the collectives that run every layer (TP, EP) on the fastest links. Let the ones that run rarely (DP, PP) cross the slowerΒ network.
Some symptoms and their usual fixes:
Symptom Usual fix------------------------------------------- ------------------------------------------------TP all-reduce dominates the timeline lower TP; keep it in-node; switch attention to DPAll-to-all dominates prefill overlap comm with compute; keep EP in-nodeAll-to-all dominates decode low-latency dispatch kernels, CUDA graphs, bigger batchOne GPU is always the slowest in MoE layers expert imbalance β EPLB / redundant expertsGPUs idle at the start/end of each step pipeline bubble β more micro-batches, interleavingOut of KV-cache memory with TP > KV heads DP-attention
For dense layers, the GPUs form aΒ grid:
total GPUs = TP x PP x DP
For MoE layers, here is the part that took me longest to see. EP is not an extra dimension of the grid. It reuses the same GPUs as TP and DP (inside each pipeline stage) and regroups them for the expertΒ weights:
TP x DP = ETP x EP x EDPETP = tensor parallelism inside each expert (usually 1 for small experts)EP = how many GPUs the experts are spread overEDP = expert data parallelism: how many copies of each expert shard exist (expert gradients are averaged only across these copies)
So a single GPU belongs to several groups at once: a TP group and a DP group for attention, an EP group and an EDP group for experts, and one pipeline stage.
Frameworks number GPUs in a fixed nesting order. Megatron-LM uses tp β cp β ep β dp β pp, with TP varying fastest. GPUs with neighbouring rank numbers sit in the same server, so TP and EP naturally stay inside a server, while DP and PP span servers. That's exactly what weΒ want.
Before the 8-GPU serving example, here is what a layout with all four strategies looks like when training thisΒ model:
PP = 2 one pipeline stage per 2 servers, 20 layers each (15 GDN + 5 full-attention)TP = 2 matches the 2 KV heads exactlyDP = 8 32 / (2 x 2)EP = 8 one server = one EP group β 32 experts per GPUEDP = 2 because TP x DP = 16 = ETP(1) x EP(8) x EDP(2)
PP stage 0 (layers 0-19) PP stage 1 (layers 20-39) ββββββββββ Node 0 ββββββββββ¬βββββββββ Node 1 βββββββββββ βββ Node 2 βββ¬ββ Node 3 βββrank: 0 1 2 3 4 5 6 7 β 8 9 10 11 12 13 14 15 16 ...... 23 24 ...... 31TP=2: [0 1][2 3][4 5][6 7] β [8 9][10 11][12 13][14 15] (same pattern)DP=8: one TP pair = one DP replica β pairs {0,1},{2,3},...,{14,15} dense DP groups: {0,2,4,...,14} and {1,3,5,...,15}EP=8: group A = {0..7} β group B = {8..15} {16..23} {24..31} r0: E0-31 r1: E32-63 r2: E64-95 ... r7: E224-255 (r8..r15 use the same split)EDP=2: (0,8) (1,9) (2,10) ... (7,15) β ranks holding the same experts, on different nodesPP p2p: rank i β rank i+16 β stage boundary, inter-node
One training step on this layout:
DP: global batch split 8 ways β each TP pair gets its own micro-batches β βΌ stage 0, for each of its 20 layers: β attention / GDN on the TP pair β reduce-scatter [2 GPUs, in-server] β router picks top-8 experts per token β all-to-all DISPATCH β 8 GPUs [in-server] β grouped GEMM on 32 local experts β all-to-all COMBINE β 8 GPUs [in-server] β all-gather for the next attention β 2 GPUs [in-server] β βΌ PP: send activations, rank i β rank i+16 [cross-server] β stage 1 runs layers 20-39 β loss β βΌ backward: same collectives in reverse (1F1B, m = 16 β bubble β 6 %) β βΌ gradient sync, overlapped with backward: [cross-server] attention / dense grads β all-reduce across DP = 8 expert grads β all-reduce across EDP = 2
The collectives that run every layer (TP and EP) never leave a server. Only the pipeline hand-off and the once-per-step gradient sync cross theΒ network.
What EP buys here:
per GPU, experts: 32 experts x 20 layers x 3.15 M β 2.0 B params β 4 GB (bf16)without EP: 256 experts x 20 layers x 3.15 M β 16 B params β 32 GB (bf16)
Now the case that started all of this: serving the model on one 8-GPUΒ server.
35 B params x 2 bytes (bf16) β 70 GB of weights
That fits comfortably across 8 GPUs, so no PP. Everything stays inside one server, which is great forΒ latency.
The simplest launch is TP = 8 for everything. This is what my original trace was:
Attention: TP = 8 β 16 q heads / 8 = 2 per GPU 2 KV heads / 8 β not possible β each KV head copied to 4 GPUsMoE: TP = 8 β every GPU holds all 256 experts, each sliced to 1/8 gate_up per expert: 2048 x 128, down: 64 x 2048
Two problems:
Attention: DP = 4 x TP = 2 β 4 independent request groups; TP = 2 matches the 2 KV headsMoE: EP = 8 β each GPU holds 32 whole expertsPP: 1
The rank layout:
One server, 8 GPUs (PP = 1) ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββrank: 0 1 2 3 4 5 6 7ATTENTION (DP = 4 x TP = 2): [ 0 1 ] [ 2 3 ] [ 4 5 ] [ 6 7 ] DP grp 0 DP grp 1 DP grp 2 DP grp 3 β different requests in each r0: kv head 0 + q heads 0-7, r1: kv head 1 + q heads 8-15 (same in every pair)MoE (EP = 8, one group spanning the whole server): r0: E0-31 r1: E32-63 r2: E64-95 r3: E96-127 r4: E128-159 r5: E160-191 r6: E192-223 r7: E224-255Same GPU, two roles: rank 5 = "TP rank 1 of DP group 2" for attention = "owner of experts 160-191" for MoE ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
KV cache per token, across the 10 full-attention layers:
2 (K and V) x 2 KV heads x 256 dim x 2 bytes x 10 layers = 20 KB per token
KV stored per token (all 8 GPUs) duplicationTP = 8 80 KB 4x (each KV head on 4 GPUs)DP = 4 x TP = 2 20 KB 1x (each request lives in one pair)
Thatβs about 4x more tokens in the KV cache, which means more concurrent requests.
Weight memory per GPU:
TP = 8 DP-attn + EP = 8experts 32.2 B / 8 β 4.0 B 32.2 B / 8 β 4.0 B β 8 GBattention / other β 2.8 B / 8 β 0.35 B β 2.8 B / 2 β 1.4 B (copied in each DP group)total weights β 8.7 GB β 10.8 GB
The EP layout spends about 2 GB more per GPU on weights, because the attention weights are copied across the 4 DP groups. In exchange it saves 4x on the KV cache, which is usually the much larger number. (The 30 GDN layers also keep a fixed-size recurrent state per sequence. Itβs split differently in the two layouts, but itβs similar in total.)
Suppose 64 requests are running, 16 in each DP group. Each request produces one token per step:
each DP group: its own batch of B_g tokens (1 per running request) B_g = 16, B_total = 64 β βΌ attention / GDN in the TP pair β all-reduce (2 GPUs) β each pair works only on its own 16 requests' KV cache β βΌ router: top-8 per token β 16 tokens x 8 = 128 (token, expert) pairs per group β βΌ all-to-all DISPATCH across all 8 GPUs (tokens from all 4 DP groups mix here) β each GPU sends β 8 tokens x 5.3 GPUs x 4 KB β 170 KB β tiny, latency-bound β βΌ each GPU: grouped GEMM on its 32 experts β rows per expert β (Ξ£ B_g) Β· 8 / 256 = B_total / 32 = 2 β experts actually hit per step β 256 Β· (1 - (1 - 1/32)^64) β 222 of 256 β βΌ all-to-all COMBINE β tokens return to their DP group β βΌ weighted sum of 8 expert outputs + residual β next layer (x 40 layers β 80 all-to-alls per generated token)
What decode tells us:
Now suppose each DP group is prefilling 4 prompts of 1,024 tokens, which is 4,096 tokens per group and 16,384 in total:
each DP group: 4 prompts x 1,024 tokens = 4,096 tokens β βΌ attention / GDN in the TP pair β all-reduce (2 GPUs) β 4,096 tokens x 4 KB = 16 MB per all-reduce β bandwidth matters now β βΌ router: top-8 per token β βΌ all-to-all DISPATCH across all 8 GPUs β each GPU sends β 2,048 tokens x 5.3 GPUs x 4 KB β 42 MB per layer β βΌ each GPU: grouped GEMM on its 32 experts β rows per expert β 16,384 x 8 / 256 = 512 β gate_up: [512 x 2048] Β· [2048 x 1024] β a real, compute-bound GEMM β down: [512 x 512] Β· [512 x 2048] β βΌ all-to-all COMBINE β same β 42 MB back
What prefill tells us:
TP = 8 DP-attn (4x2) + EP = 8---------------------------- ------------------------- ---------------------------attention split 2 q heads, KV copied 4x 8 q heads, 1 KV head, no copyKV cache per token (total) 80 KB 20 KBexperts per GPU 256 slices (1/8 each) 32 whole expertsexpert GEMM shape (gate_up) 2048 x 128 2048 x 1024expert weight bytes per GPU same samerows per expert (same batch) same sameMoE collective all-reduce of all tokens 2 x all-to-all, routed onlyload balance perfect router-dependentweights per GPU β 8.7 GB β 10.8 GBmain win simplicity, balance 4x KV room β bigger batches, better expert GEMM shapes
If you need more throughput, there are two directions:
PP only enters the picture if a single replica canβt fit in one server, which isnβt the case for this model.
What I originally saw in the trace (every GPU running thin slices of all 256 experts) wasnβt wrong. It was simply TP = 8 applied everywhere. Once you see the four strategies as four different ways of splitting the work, the better layout for this model is clear.
DP, TP, PP and EP: How LLM Parallelism Actually Fits Together was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.