{"slug": "dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together", "title": "DP, TP, PP and EP: How LLM Parallelism Actually Fits Together", "summary": "A GPU trace of a Mixture-of-Experts model served on 8 GPUs showed every GPU running all 256 experts as thin 1/8 slices rather than distributing experts across devices, an analysis of Qwen3.5-35B-A3B explains. In that model, roughly 32.2 billion of the 35 billion parameters are experts (256 experts × 3.15 million params × 40 layers), while each token touches only 8 of 256 experts per layer, so memory is dominated by experts and per-token compute stays small. The breakdown compares data, tensor, pipeline and expert parallelism by what each communicates and how often, with tensor and expert parallelism on the critical path every layer while data parallelism's gradient all-reduce runs once per training step.", "body_md": "I was looking at a GPU trace of a Mixture-of-Experts model served on 8 GPUs, and one thing didn’t make sense to me.\n\nThe model has 256 experts. I had always pictured MoE serving with the experts spread out, a few dozen on each GPU, and tokens travelling to wherever their experts live. That’s not what the trace showed. **Every GPU was running all 256 experts, and each one was a thin 1/8 slice of the real expert.** The matrix multiplications were narrow, and there were a lot of them.\n\nThat raised a set of questions I couldn’t answer:\n\nThis post is what I found. I’ll explain each kind of parallelism, when you need it and how they combine, then work through one real model on 8 GPUs with actual numbers.\n\n**Qwen3.5–35B-A3B**: about 35B parameters in total, about 3B active per token.\n\n```\nLayers          40  (30 linear-attention \"GDN\" layers + 10 full-attention layers)Attention       16 query heads, 2 KV heads, head dim 256MoE             every layer: 256 experts, each token uses the top 8Hidden size     H = 2048Expert MLP      SwiGLU, intermediate size I = 512                  gate_up: 2048 x 1024   (2.10 M params)                  down:    512  x 2048   (1.05 M params)                  ≈ 3.15 M params per expert\n```\n\nOne number explains most of what follows:\n\n```\n256 experts x 3.15 M params x 40 layers ≈ 32.2 B params\n```\n\nSo **about 32 of the 35 billion parameters are experts**, yet each token touches only 8 of the 256 experts per layer. Memory is dominated by experts, while compute per token is small. Keep that in mind.\n\nEach form of parallelism answers one question: **which part of the work do we split across GPUs?**\n\n```\n                          ┌── the batch ──────────► Data Parallel     (DP)  the work of one step ───┼── the weight matrices ► Tensor Parallel   (TP)                          ├── the layers ─────────► Pipeline Parallel (PP)                          └── the experts ────────► Expert Parallel   (EP, MoE only)\n```\n\nThere are also a few variants you’ll see in configs:\n\nThe four core strategies differ mainly in **what they communicate, and how often**:\n\n```\nStrategy   Collective                     How often                    On the critical path?--------   ----------------------------   --------------------------   ---------------------DP         all-reduce of gradients        once per training step       no (overlapped)TP         all-reduce of activations      every layer                  YESPP         point-to-point send/recv       once per stage boundary      partly (idle \"bubble\")EP         all-to-all of routed tokens    twice per MoE layer          YES\n```\n\nAlmost every layout decision follows from this table.\n\nEach GPU holds a **full copy of the model** and processes different samples.\n\n```\nglobal batch ──┬──► GPU 0: full model ──► gradients₀ ──┐               ├──► GPU 1: full model ──► gradients₁ ──┼──► all-reduce (average)               ├──► GPU 2: full model ──► gradients₂ ──┤        │               └──► GPU 3: full model ──► gradients₃ ──┘        ▼                                                     every GPU applies the same update\n```\n\nTraining memory with mixed-precision Adam is about 16 bytes per parameter: 2 for bf16 weights, 2 for bf16 gradients and 12 for the fp32 master copy plus Adam’s two moments. ZeRO shards these across the DP group:\n\n```\nPlain DP   16 B/param on every GPUZeRO-1     4 + 12/DP      (shard optimizer states)ZeRO-2     2 + 14/DP      (also shard gradients)ZeRO-3     16/DP          (also shard weights; all-gather them layer by layer)\n```\n\nTP splits **each layer’s weight matrices** across GPUs so they compute every layer together. The standard trick (from Megatron-LM) pairs a column split with a row split so that only **one** sync is needed per block:\n\n```\nMLP:  Y = act(X · A) · B          with TP = 2  A split by COLUMNS:  A = [A₀ | A₁]      GPU r computes  Zᵣ = act(X · Aᵣ)    ← no communicationB split by ROWS:     B = [B₀ ; B₁]      GPU r computes  Yᵣ = Zᵣ · Bᵣ                                          Y = Y₀ + Y₁                       ← one all-reduce\n```\n\nAttention works the same way. Q/K/V are split **by heads**, so each GPU owns a subset of heads, and the output projection is row-split and followed by one all-reduce.\n\nFor Qwen3.5–35B-A3B with TP = 2, each GPU owns 8 of the 16 query heads and 1 of the 2 KV heads.\n\n**Cost.** Every all-reduce moves the full activation, T tokens x 2048 x 2 bytes, which is **4 KB per token per all-reduce**. It happens every layer, and nothing can proceed until it finishes. That is why TP needs very fast GPU-to-GPU links (NVLink, xGMI and similar) and almost always stays **inside one server**.\n\n**The KV-head limit.** This one surprised me. TP splits attention by KV heads, so it can’t usefully go beyond the number of KV heads:\n\n```\nQwen3.5-35B-A3B has 2 KV heads.TP = 2   →  GPU 0 gets KV head 0, GPU 1 gets KV head 1        clean splitTP = 8   →  8 GPUs, only 2 KV heads → each head copied to 4 GPUs            → the KV cache is stored 4 times\n```\n\nKeep this in mind. It’s the main reason the 8-GPU example below doesn’t simply use TP = 8.\n\n**Sequence Parallelism (SP)** comes paired with TP. Outside the matmuls (LayerNorm, residual adds), activations are split along the sequence instead of being copied. The all-reduce becomes a reduce-scatter plus an all-gather. The traffic is the same, but activation memory per GPU drops by the TP factor.\n\nEach GPU (or group of GPUs) owns a contiguous block of layers, called a **stage**. Activations flow forward through the stages and gradients flow back.\n\n```\nstage 0: layers 0-19  ──activations──►  stage 1: layers 20-39                      ◄──gradients────\n```\n\nThe problem is the **bubble**. With a single batch, stage 1 waits while stage 0 works, and the reverse happens on the way back. The fix is to cut the batch into many **micro-batches** and interleave them:\n\n```\n1F1B schedule, 2 stages, 4 micro-batches    (F = forward, B = backward, .. = idle) stage 0:  F1 F2 F3 F4 .. B1 B2 B3 B4 stage 1:  .. F1 B1 F2 B2 F3 B3 F4 B4           ^^                      ^^   bubble\nbubble fraction = (p - 1) / (m + p - 1)        p = stages, m = micro-batchesp = 2, m = 16   →   1/17 ≈ 6 %\n```\n\nIn a MoE layer, the dense MLP is replaced by many expert MLPs plus a small **router**. The router picks the top-k experts for each token, and the token’s output is a weighted sum of what those experts produce:\n\n```\ny = Σ  gᵢ(x) · Expertᵢ(x)      over the top-8 experts chosen for token x\n```\n\nEP gives each GPU **a subset of whole experts**. With 256 experts and EP = 8, that’s 32 experts per GPU. Tokens are then sent to wherever their experts live:\n\n```\n     GPU 0            GPU 1                       GPU 7  its tokens        its tokens       ...        its tokens      │                 │                           │   router            router                      router        top-8 of 256 per token      │                 │                           │  ════════════════ all-to-all DISPATCH ══════════════════════   token → GPU (expert_id ÷ 32)      │                 │                           │  experts 0-31     experts 32-63     ...      experts 224-255   one grouped GEMM per GPU      │                 │                           │  ════════════════ all-to-all COMBINE  ══════════════════════   results go back home      │                 │                           │  weighted sum      weighted sum                weighted sum\n```\n\n**Cost.** There are two all-to-alls per MoE layer. A token is 4 KB (2048 values in bf16). Good kernels (for example DeepEP) send one copy per destination **GPU**, not per expert. With 8 experts out of 256 spread across 8 GPUs, a token lands on about **5.3 distinct GPUs** on average:\n\n```\ndistinct GPUs per token = EP · (1 - C(E - E/EP, k) / C(E, k))                        = 8  · (1 - C(224, 8) / C(256, 8))                        ≈ 8  · (1 - 0.338)  ≈ 5.3\n```\n\n**The weakness is load imbalance.** If the router sends a lot of tokens to a few popular experts, the GPUs holding them become stragglers, and everyone waits at the combine step. The usual fixes:\n\n**EP versus TP on the experts.** This was my original question. Without EP, frameworks apply TP to the MoE layers too: every GPU holds **all** experts, each one sliced. That’s exactly what my trace showed.\n\n```\n                               TP = 8 on experts          EP = 8                               ------------------         ------------------experts per GPU                256 (each 1/8 slice)       32 (whole)gate_up GEMM per expert        2048 x 128                 2048 x 1024down GEMM per expert           64   x 2048                512  x 2048expert weight bytes per GPU    1/8 of total               1/8 of total     ← same!collective per MoE layer       all-reduce, all tokens     2 x all-to-all, routed tokens onlyload balance                   perfect                    depends on router\n```\n\nFor a fixed batch, the **expert weight bytes per GPU are the same**, and so is the average number of rows each expert processes. What EP changes is:\n\nA practical way to decide, in order:\n\n```\n1. Does the model (plus optimizer / KV cache) fit on one GPU?      YES → use DP only. Done.  (training: add ZeRO if optimizer states don't fit)2. Is it a MoE model?      YES → use EP for the expert weights. Experts dominate memory, and EP divides them.            Keep the EP group inside one server if possible (all-to-all is latency-critical).3. Attention / dense layers still too big, or need lower latency?      → use TP inside one server, up to the number of KV heads.      → beyond that, use DP-attention instead of more TP (avoids copying the KV cache).4. Still doesn't fit, or the job spans several servers?      → add PP across servers (cheap on bandwidth). Use many micro-batches to keep        the bubble small.5. Very long sequences (100K+ tokens)?      → add CP to split the sequence for attention.6. Fill whatever GPUs remain with DP (and, for experts, expert-DP).\n```\n\nThe rule underneath all of this:\n\n***Put the collectives that run every layer (TP, EP) on the fastest links. Let the ones that run rarely (DP, PP) cross the slower network.***\n\nSome symptoms and their usual fixes:\n\n```\nSymptom                                       Usual fix-------------------------------------------   ------------------------------------------------TP all-reduce dominates the timeline          lower TP; keep it in-node; switch attention to DPAll-to-all dominates prefill                  overlap comm with compute; keep EP in-nodeAll-to-all dominates decode                   low-latency dispatch kernels, CUDA graphs, bigger batchOne GPU is always the slowest in MoE layers   expert imbalance → EPLB / redundant expertsGPUs idle at the start/end of each step       pipeline bubble → more micro-batches, interleavingOut of KV-cache memory with TP > KV heads     DP-attention\n```\n\nFor **dense layers**, the GPUs form a grid:\n\n```\ntotal GPUs = TP x PP x DP\n```\n\nFor **MoE layers**, here is the part that took me longest to see. **EP is not an extra dimension of the grid.** It reuses the same GPUs as TP and DP (inside each pipeline stage) and regroups them for the expert weights:\n\n```\nTP x DP  =  ETP x EP x EDPETP  = tensor parallelism inside each expert (usually 1 for small experts)EP   = how many GPUs the experts are spread overEDP  = expert data parallelism: how many copies of each expert shard exist       (expert gradients are averaged only across these copies)\n```\n\nSo a single GPU belongs to several groups at once: a TP group and a DP group for attention, an EP group and an EDP group for experts, and one pipeline stage.\n\nFrameworks number GPUs in a fixed nesting order. Megatron-LM uses tp → cp → ep → dp → pp, with TP varying fastest. GPUs with neighbouring rank numbers sit in the same server, so **TP and EP naturally stay inside a server, while DP and PP span servers.** That's exactly what we want.\n\nBefore the 8-GPU serving example, here is what a layout with **all four** strategies looks like when training this model:\n\n```\nPP = 2    one pipeline stage per 2 servers, 20 layers each (15 GDN + 5 full-attention)TP = 2    matches the 2 KV heads exactlyDP = 8    32 / (2 x 2)EP = 8    one server = one EP group → 32 experts per GPUEDP = 2   because TP x DP = 16 = ETP(1) x EP(8) x EDP(2)\nPP stage 0 (layers 0-19)                     PP stage 1 (layers 20-39)        ┌───────── Node 0 ─────────┬───────── Node 1 ──────────┐ ┌── Node 2 ──┬── Node 3 ──┐rank:     0  1  2  3  4  5  6  7   │  8  9 10 11 12 13 14 15     16 ...... 23  24 ...... 31TP=2:    [0 1][2 3][4 5][6 7]      │ [8 9][10 11][12 13][14 15]    (same pattern)DP=8:    one TP pair = one DP replica → pairs {0,1},{2,3},...,{14,15}         dense DP groups: {0,2,4,...,14} and {1,3,5,...,15}EP=8:    group A = {0..7}          │ group B = {8..15}             {16..23}    {24..31}         r0: E0-31   r1: E32-63   r2: E64-95  ...  r7: E224-255   (r8..r15 use the same split)EDP=2:   (0,8) (1,9) (2,10) ... (7,15)   ← ranks holding the same experts, on different nodesPP p2p:  rank i ⇄ rank i+16              ← stage boundary, inter-node\n```\n\nOne training step on this layout:\n\n```\nDP: global batch split 8 ways → each TP pair gets its own micro-batches │ ▼ stage 0, for each of its 20 layers: │    attention / GDN on the TP pair          → reduce-scatter   [2 GPUs, in-server] │    router picks top-8 experts per token │    all-to-all DISPATCH                     → 8 GPUs           [in-server] │    grouped GEMM on 32 local experts │    all-to-all COMBINE                      → 8 GPUs           [in-server] │    all-gather for the next attention       → 2 GPUs           [in-server] │ ▼ PP: send activations, rank i → rank i+16                      [cross-server] │ stage 1 runs layers 20-39 → loss │ ▼ backward: same collectives in reverse (1F1B, m = 16 → bubble ≈ 6 %) │ ▼ gradient sync, overlapped with backward:                      [cross-server]      attention / dense grads → all-reduce across DP  = 8      expert grads            → all-reduce across EDP = 2\n```\n\nThe collectives that run **every layer** (TP and EP) never leave a server. Only the pipeline hand-off and the once-per-step gradient sync cross the network.\n\nWhat EP buys here:\n\n```\nper GPU, experts:   32 experts x 20 layers x 3.15 M ≈ 2.0 B params ≈  4 GB (bf16)without EP:        256 experts x 20 layers x 3.15 M ≈ 16  B params ≈ 32 GB (bf16)\n```\n\nNow the case that started all of this: **serving the model on one 8-GPU server.**\n\n```\n35 B params x 2 bytes (bf16) ≈ 70 GB of weights\n```\n\nThat fits comfortably across 8 GPUs, so **no PP**. Everything stays inside one server, which is great for latency.\n\nThe simplest launch is TP = 8 for everything. This is what my original trace was:\n\n```\nAttention:  TP = 8   → 16 q heads / 8 = 2 per GPU                       2 KV heads / 8 → not possible → each KV head copied to 4 GPUsMoE:        TP = 8   → every GPU holds all 256 experts, each sliced to 1/8                       gate_up per expert: 2048 x 128,  down: 64 x 2048\n```\n\nTwo problems:\n\n```\nAttention:  DP = 4 x TP = 2   → 4 independent request groups; TP = 2 matches the 2 KV headsMoE:        EP = 8            → each GPU holds 32 whole expertsPP:         1\n```\n\nThe rank layout:\n\n```\nOne server, 8 GPUs (PP = 1)        ┌──────────────────────────────────────────────────────────────────┐rank:      0     1     2     3     4     5     6     7ATTENTION (DP = 4 x TP = 2):           [ 0  1 ]    [ 2  3 ]    [ 4  5 ]    [ 6  7 ]            DP grp 0    DP grp 1    DP grp 2    DP grp 3   ← different requests in each            r0: kv head 0 + q heads 0-7,  r1: kv head 1 + q heads 8-15  (same in every pair)MoE (EP = 8, one group spanning the whole server):           r0: E0-31     r1: E32-63    r2: E64-95    r3: E96-127           r4: E128-159  r5: E160-191  r6: E192-223  r7: E224-255Same GPU, two roles:  rank 5 = \"TP rank 1 of DP group 2\" for attention                             = \"owner of experts 160-191\" for MoE        └──────────────────────────────────────────────────────────────────┘\n```\n\nKV cache per token, across the 10 full-attention layers:\n\n```\n2 (K and V) x 2 KV heads x 256 dim x 2 bytes x 10 layers = 20 KB per token\nKV stored per token (all 8 GPUs)    duplicationTP = 8                   80 KB                               4x   (each KV head on 4 GPUs)DP = 4 x TP = 2          20 KB                               1x   (each request lives in one pair)\n```\n\nThat’s **about 4x more tokens in the KV cache**, which means more concurrent requests.\n\nWeight memory per GPU:\n\n```\nTP = 8                    DP-attn + EP = 8experts                   32.2 B / 8 ≈ 4.0 B        32.2 B / 8 ≈ 4.0 B       ≈ 8 GBattention / other         ≈ 2.8 B / 8 ≈ 0.35 B      ≈ 2.8 B / 2 ≈ 1.4 B      (copied in each DP group)total weights             ≈ 8.7 GB                  ≈ 10.8 GB\n```\n\nThe EP layout spends about 2 GB more per GPU on weights, because the attention weights are copied across the 4 DP groups. In exchange it saves 4x on the KV cache, which is usually the much larger number. (The 30 GDN layers also keep a fixed-size recurrent state per sequence. It’s split differently in the two layouts, but it’s similar in total.)\n\nSuppose 64 requests are running, 16 in each DP group. Each request produces one token per step:\n\n```\neach DP group: its own batch of B_g tokens (1 per running request)     B_g = 16, B_total = 64   │   ▼ attention / GDN in the TP pair → all-reduce (2 GPUs)   │       each pair works only on its own 16 requests' KV cache   │   ▼ router: top-8 per token   │       16 tokens x 8 = 128 (token, expert) pairs per group   │   ▼ all-to-all DISPATCH across all 8 GPUs  (tokens from all 4 DP groups mix here)   │       each GPU sends ≈ 8 tokens x 5.3 GPUs x 4 KB ≈ 170 KB  → tiny, latency-bound   │   ▼ each GPU: grouped GEMM on its 32 experts   │       rows per expert ≈ (Σ B_g) · 8 / 256 = B_total / 32 = 2   │       experts actually hit per step ≈ 256 · (1 - (1 - 1/32)^64) ≈ 222 of 256   │   ▼ all-to-all COMBINE → tokens return to their DP group   │   ▼ weighted sum of 8 expert outputs + residual → next layer         (x 40 layers → 80 all-to-alls per generated token)\n```\n\nWhat decode tells us:\n\nNow suppose each DP group is prefilling 4 prompts of 1,024 tokens, which is 4,096 tokens per group and 16,384 in total:\n\n```\neach DP group: 4 prompts x 1,024 tokens = 4,096 tokens   │   ▼ attention / GDN in the TP pair → all-reduce (2 GPUs)   │       4,096 tokens x 4 KB = 16 MB per all-reduce → bandwidth matters now   │   ▼ router: top-8 per token   │   ▼ all-to-all DISPATCH across all 8 GPUs   │       each GPU sends ≈ 2,048 tokens x 5.3 GPUs x 4 KB ≈ 42 MB per layer   │   ▼ each GPU: grouped GEMM on its 32 experts   │       rows per expert ≈ 16,384 x 8 / 256 = 512   │       gate_up: [512 x 2048] · [2048 x 1024]   ← a real, compute-bound GEMM   │       down:    [512 x 512]  · [512 x 2048]   │   ▼ all-to-all COMBINE → same ≈ 42 MB back\n```\n\nWhat prefill tells us:\n\n```\n                                TP = 8                      DP-attn (4x2) + EP = 8----------------------------    -------------------------   ---------------------------attention split                 2 q heads, KV copied 4x     8 q heads, 1 KV head, no copyKV cache per token (total)      80 KB                       20 KBexperts per GPU                 256 slices (1/8 each)       32 whole expertsexpert GEMM shape (gate_up)     2048 x 128                  2048 x 1024expert weight bytes per GPU     same                        samerows per expert (same batch)    same                        sameMoE collective                  all-reduce of all tokens    2 x all-to-all, routed onlyload balance                    perfect                     router-dependentweights per GPU                 ≈ 8.7 GB                    ≈ 10.8 GBmain win                        simplicity, balance         4x KV room → bigger batches,                                                            better expert GEMM shapes\n```\n\nIf you need more throughput, there are two directions:\n\nPP only enters the picture if a single replica can’t fit in one server, which isn’t the case for this model.\n\nWhat I originally saw in the trace (every GPU running thin slices of all 256 experts) wasn’t wrong. It was simply TP = 8 applied everywhere. Once you see the four strategies as four different ways of splitting the work, the better layout for this model is clear.\n\n[DP, TP, PP and EP: How LLM Parallelism Actually Fits Together](https://pub.towardsai.net/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together-2cc4e611da72) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together", "canonical_source": "https://pub.towardsai.net/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together-2cc4e611da72?source=rss----98111c9905da---4", "published_at": "2026-11-22 06:53:20+00:00", "updated_at": "2026-11-22 07:13:50.600579+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "large-language-models"], "entities": ["Qwen3.5-35B-A3B", "Megatron-LM", "ZeRO"], "also_reported_by": [], "takeaway": "In Qwen3.5-35B-A3B, about 32.2B of 35B parameters are experts (256 experts × 3.15M params × 40 layers) while each token uses only top-8 experts per layer, so expert memory dominates and tensor plus expert parallelism sit on the critical path every layer.", "recommendation": {"advice": "Plan MoE serving around expert weight memory: with 256 experts at 3.15M params each across 40 layers, shard experts and weights rather than replicating them, since per-token compute touches only 8 experts per layer.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together", "markdown": "https://wpnews.pro/news/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together.md", "text": "https://wpnews.pro/news/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together.txt", "jsonld": "https://wpnews.pro/news/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together.jsonld"}}