# DP, TP, PP and EP: How LLM Parallelism Actually Fits Together

> Source: <https://pub.towardsai.net/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together-2cc4e611da72?source=rss----98111c9905da---4>
> Published: 2026-11-22 06:53:20+00:00

**Recommendation:** Plan MoE serving around expert weight memory: with 256 experts at 3.15M params each across 40 layers, shard experts and weights rather than replicating them, since per-token compute touches only 8 experts per layer.
More in the playbook "Inference and self-hosting": <https://wpnews.pro/playbooks/inference-serving>

I was looking at a GPU trace of a Mixture-of-Experts model served on 8 GPUs, and one thing didn’t make sense to me.

The model has 256 experts. I had always pictured MoE serving with the experts spread out, a few dozen on each GPU, and tokens travelling to wherever their experts live. That’s not what the trace showed. **Every GPU was running all 256 experts, and each one was a thin 1/8 slice of the real expert.** The matrix multiplications were narrow, and there were a lot of them.

That raised a set of questions I couldn’t answer:

This post is what I found. I’ll explain each kind of parallelism, when you need it and how they combine, then work through one real model on 8 GPUs with actual numbers.

**Qwen3.5–35B-A3B**: about 35B parameters in total, about 3B active per token.

```
Layers          40  (30 linear-attention "GDN" layers + 10 full-attention layers)Attention       16 query heads, 2 KV heads, head dim 256MoE             every layer: 256 experts, each token uses the top 8Hidden size     H = 2048Expert MLP      SwiGLU, intermediate size I = 512                  gate_up: 2048 x 1024   (2.10 M params)                  down:    512  x 2048   (1.05 M params)                  ≈ 3.15 M params per expert
```

One number explains most of what follows:

```
256 experts x 3.15 M params x 40 layers ≈ 32.2 B params
```

So **about 32 of the 35 billion parameters are experts**, yet each token touches only 8 of the 256 experts per layer. Memory is dominated by experts, while compute per token is small. Keep that in mind.

Each form of parallelism answers one question: **which part of the work do we split across GPUs?**

```
                          ┌── the batch ──────────► Data Parallel     (DP)  the work of one step ───┼── the weight matrices ► Tensor Parallel   (TP)                          ├── the layers ─────────► Pipeline Parallel (PP)                          └── the experts ────────► Expert Parallel   (EP, MoE only)
```

There are also a few variants you’ll see in configs:

The four core strategies differ mainly in **what they communicate, and how often**:

```
Strategy   Collective                     How often                    On the critical path?--------   ----------------------------   --------------------------   ---------------------DP         all-reduce of gradients        once per training step       no (overlapped)TP         all-reduce of activations      every layer                  YESPP         point-to-point send/recv       once per stage boundary      partly (idle "bubble")EP         all-to-all of routed tokens    twice per MoE layer          YES
```

Almost every layout decision follows from this table.

Each GPU holds a **full copy of the model** and processes different samples.

```
global batch ──┬──► GPU 0: full model ──► gradients₀ ──┐               ├──► GPU 1: full model ──► gradients₁ ──┼──► all-reduce (average)               ├──► GPU 2: full model ──► gradients₂ ──┤        │               └──► GPU 3: full model ──► gradients₃ ──┘        ▼                                                     every GPU applies the same update
```

Training memory with mixed-precision Adam is about 16 bytes per parameter: 2 for bf16 weights, 2 for bf16 gradients and 12 for the fp32 master copy plus Adam’s two moments. ZeRO shards these across the DP group:

```
Plain DP   16 B/param on every GPUZeRO-1     4 + 12/DP      (shard optimizer states)ZeRO-2     2 + 14/DP      (also shard gradients)ZeRO-3     16/DP          (also shard weights; all-gather them layer by layer)
```

TP splits **each layer’s weight matrices** across GPUs so they compute every layer together. The standard trick (from Megatron-LM) pairs a column split with a row split so that only **one** sync is needed per block:

```
MLP:  Y = act(X · A) · B          with TP = 2  A split by COLUMNS:  A = [A₀ | A₁]      GPU r computes  Zᵣ = act(X · Aᵣ)    ← no communicationB split by ROWS:     B = [B₀ ; B₁]      GPU r computes  Yᵣ = Zᵣ · Bᵣ                                          Y = Y₀ + Y₁                       ← one all-reduce
```

Attention works the same way. Q/K/V are split **by heads**, so each GPU owns a subset of heads, and the output projection is row-split and followed by one all-reduce.

For Qwen3.5–35B-A3B with TP = 2, each GPU owns 8 of the 16 query heads and 1 of the 2 KV heads.

**Cost.** Every all-reduce moves the full activation, T tokens x 2048 x 2 bytes, which is **4 KB per token per all-reduce**. It happens every layer, and nothing can proceed until it finishes. That is why TP needs very fast GPU-to-GPU links (NVLink, xGMI and similar) and almost always stays **inside one server**.

**The KV-head limit.** This one surprised me. TP splits attention by KV heads, so it can’t usefully go beyond the number of KV heads:

```
Qwen3.5-35B-A3B has 2 KV heads.TP = 2   →  GPU 0 gets KV head 0, GPU 1 gets KV head 1        clean splitTP = 8   →  8 GPUs, only 2 KV heads → each head copied to 4 GPUs            → the KV cache is stored 4 times
```

Keep this in mind. It’s the main reason the 8-GPU example below doesn’t simply use TP = 8.

**Sequence Parallelism (SP)** comes paired with TP. Outside the matmuls (LayerNorm, residual adds), activations are split along the sequence instead of being copied. The all-reduce becomes a reduce-scatter plus an all-gather. The traffic is the same, but activation memory per GPU drops by the TP factor.

Each GPU (or group of GPUs) owns a contiguous block of layers, called a **stage**. Activations flow forward through the stages and gradients flow back.

```
stage 0: layers 0-19  ──activations──►  stage 1: layers 20-39                      ◄──gradients────
```

The problem is the **bubble**. With a single batch, stage 1 waits while stage 0 works, and the reverse happens on the way back. The fix is to cut the batch into many **micro-batches** and interleave them:

```
1F1B schedule, 2 stages, 4 micro-batches    (F = forward, B = backward, .. = idle) stage 0:  F1 F2 F3 F4 .. B1 B2 B3 B4 stage 1:  .. F1 B1 F2 B2 F3 B3 F4 B4           ^^                      ^^   bubble
bubble fraction = (p - 1) / (m + p - 1)        p = stages, m = micro-batchesp = 2, m = 16   →   1/17 ≈ 6 %
```

In a MoE layer, the dense MLP is replaced by many expert MLPs plus a small **router**. The router picks the top-k experts for each token, and the token’s output is a weighted sum of what those experts produce:

```
y = Σ  gᵢ(x) · Expertᵢ(x)      over the top-8 experts chosen for token x
```

EP gives each GPU **a subset of whole experts**. With 256 experts and EP = 8, that’s 32 experts per GPU. Tokens are then sent to wherever their experts live:

```
     GPU 0            GPU 1                       GPU 7  its tokens        its tokens       ...        its tokens      │                 │                           │   router            router                      router        top-8 of 256 per token      │                 │                           │  ════════════════ all-to-all DISPATCH ══════════════════════   token → GPU (expert_id ÷ 32)      │                 │                           │  experts 0-31     experts 32-63     ...      experts 224-255   one grouped GEMM per GPU      │                 │                           │  ════════════════ all-to-all COMBINE  ══════════════════════   results go back home      │                 │                           │  weighted sum      weighted sum                weighted sum
```

**Cost.** There are two all-to-alls per MoE layer. A token is 4 KB (2048 values in bf16). Good kernels (for example DeepEP) send one copy per destination **GPU**, not per expert. With 8 experts out of 256 spread across 8 GPUs, a token lands on about **5.3 distinct GPUs** on average:

```
distinct GPUs per token = EP · (1 - C(E - E/EP, k) / C(E, k))                        = 8  · (1 - C(224, 8) / C(256, 8))                        ≈ 8  · (1 - 0.338)  ≈ 5.3
```

**The weakness is load imbalance.** If the router sends a lot of tokens to a few popular experts, the GPUs holding them become stragglers, and everyone waits at the combine step. The usual fixes:

**EP versus TP on the experts.** This was my original question. Without EP, frameworks apply TP to the MoE layers too: every GPU holds **all** experts, each one sliced. That’s exactly what my trace showed.

```
                               TP = 8 on experts          EP = 8                               ------------------         ------------------experts per GPU                256 (each 1/8 slice)       32 (whole)gate_up GEMM per expert        2048 x 128                 2048 x 1024down GEMM per expert           64   x 2048                512  x 2048expert weight bytes per GPU    1/8 of total               1/8 of total     ← same!collective per MoE layer       all-reduce, all tokens     2 x all-to-all, routed tokens onlyload balance                   perfect                    depends on router
```

For a fixed batch, the **expert weight bytes per GPU are the same**, and so is the average number of rows each expert processes. What EP changes is:

A practical way to decide, in order:

```
1. Does the model (plus optimizer / KV cache) fit on one GPU?      YES → use DP only. Done.  (training: add ZeRO if optimizer states don't fit)2. Is it a MoE model?      YES → use EP for the expert weights. Experts dominate memory, and EP divides them.            Keep the EP group inside one server if possible (all-to-all is latency-critical).3. Attention / dense layers still too big, or need lower latency?      → use TP inside one server, up to the number of KV heads.      → beyond that, use DP-attention instead of more TP (avoids copying the KV cache).4. Still doesn't fit, or the job spans several servers?      → add PP across servers (cheap on bandwidth). Use many micro-batches to keep        the bubble small.5. Very long sequences (100K+ tokens)?      → add CP to split the sequence for attention.6. Fill whatever GPUs remain with DP (and, for experts, expert-DP).
```

The rule underneath all of this:

***Put the collectives that run every layer (TP, EP) on the fastest links. Let the ones that run rarely (DP, PP) cross the slower network.***

Some symptoms and their usual fixes:

```
Symptom                                       Usual fix-------------------------------------------   ------------------------------------------------TP all-reduce dominates the timeline          lower TP; keep it in-node; switch attention to DPAll-to-all dominates prefill                  overlap comm with compute; keep EP in-nodeAll-to-all dominates decode                   low-latency dispatch kernels, CUDA graphs, bigger batchOne GPU is always the slowest in MoE layers   expert imbalance → EPLB / redundant expertsGPUs idle at the start/end of each step       pipeline bubble → more micro-batches, interleavingOut of KV-cache memory with TP > KV heads     DP-attention
```

For **dense layers**, the GPUs form a grid:

```
total GPUs = TP x PP x DP
```

For **MoE layers**, here is the part that took me longest to see. **EP is not an extra dimension of the grid.** It reuses the same GPUs as TP and DP (inside each pipeline stage) and regroups them for the expert weights:

```
TP x DP  =  ETP x EP x EDPETP  = tensor parallelism inside each expert (usually 1 for small experts)EP   = how many GPUs the experts are spread overEDP  = expert data parallelism: how many copies of each expert shard exist       (expert gradients are averaged only across these copies)
```

So a single GPU belongs to several groups at once: a TP group and a DP group for attention, an EP group and an EDP group for experts, and one pipeline stage.

Frameworks number GPUs in a fixed nesting order. Megatron-LM uses tp → cp → ep → dp → pp, with TP varying fastest. GPUs with neighbouring rank numbers sit in the same server, so **TP and EP naturally stay inside a server, while DP and PP span servers.** That's exactly what we want.

Before the 8-GPU serving example, here is what a layout with **all four** strategies looks like when training this model:

```
PP = 2    one pipeline stage per 2 servers, 20 layers each (15 GDN + 5 full-attention)TP = 2    matches the 2 KV heads exactlyDP = 8    32 / (2 x 2)EP = 8    one server = one EP group → 32 experts per GPUEDP = 2   because TP x DP = 16 = ETP(1) x EP(8) x EDP(2)
PP stage 0 (layers 0-19)                     PP stage 1 (layers 20-39)        ┌───────── Node 0 ─────────┬───────── Node 1 ──────────┐ ┌── Node 2 ──┬── Node 3 ──┐rank:     0  1  2  3  4  5  6  7   │  8  9 10 11 12 13 14 15     16 ...... 23  24 ...... 31TP=2:    [0 1][2 3][4 5][6 7]      │ [8 9][10 11][12 13][14 15]    (same pattern)DP=8:    one TP pair = one DP replica → pairs {0,1},{2,3},...,{14,15}         dense DP groups: {0,2,4,...,14} and {1,3,5,...,15}EP=8:    group A = {0..7}          │ group B = {8..15}             {16..23}    {24..31}         r0: E0-31   r1: E32-63   r2: E64-95  ...  r7: E224-255   (r8..r15 use the same split)EDP=2:   (0,8) (1,9) (2,10) ... (7,15)   ← ranks holding the same experts, on different nodesPP p2p:  rank i ⇄ rank i+16              ← stage boundary, inter-node
```

One training step on this layout:

```
DP: global batch split 8 ways → each TP pair gets its own micro-batches │ ▼ stage 0, for each of its 20 layers: │    attention / GDN on the TP pair          → reduce-scatter   [2 GPUs, in-server] │    router picks top-8 experts per token │    all-to-all DISPATCH                     → 8 GPUs           [in-server] │    grouped GEMM on 32 local experts │    all-to-all COMBINE                      → 8 GPUs           [in-server] │    all-gather for the next attention       → 2 GPUs           [in-server] │ ▼ PP: send activations, rank i → rank i+16                      [cross-server] │ stage 1 runs layers 20-39 → loss │ ▼ backward: same collectives in reverse (1F1B, m = 16 → bubble ≈ 6 %) │ ▼ gradient sync, overlapped with backward:                      [cross-server]      attention / dense grads → all-reduce across DP  = 8      expert grads            → all-reduce across EDP = 2
```

The collectives that run **every layer** (TP and EP) never leave a server. Only the pipeline hand-off and the once-per-step gradient sync cross the network.

What EP buys here:

```
per GPU, experts:   32 experts x 20 layers x 3.15 M ≈ 2.0 B params ≈  4 GB (bf16)without EP:        256 experts x 20 layers x 3.15 M ≈ 16  B params ≈ 32 GB (bf16)
```

Now the case that started all of this: **serving the model on one 8-GPU server.**

```
35 B params x 2 bytes (bf16) ≈ 70 GB of weights
```

That fits comfortably across 8 GPUs, so **no PP**. Everything stays inside one server, which is great for latency.

The simplest launch is TP = 8 for everything. This is what my original trace was:

```
Attention:  TP = 8   → 16 q heads / 8 = 2 per GPU                       2 KV heads / 8 → not possible → each KV head copied to 4 GPUsMoE:        TP = 8   → every GPU holds all 256 experts, each sliced to 1/8                       gate_up per expert: 2048 x 128,  down: 64 x 2048
```

Two problems:

```
Attention:  DP = 4 x TP = 2   → 4 independent request groups; TP = 2 matches the 2 KV headsMoE:        EP = 8            → each GPU holds 32 whole expertsPP:         1
```

The rank layout:

```
One server, 8 GPUs (PP = 1)        ┌──────────────────────────────────────────────────────────────────┐rank:      0     1     2     3     4     5     6     7ATTENTION (DP = 4 x TP = 2):           [ 0  1 ]    [ 2  3 ]    [ 4  5 ]    [ 6  7 ]            DP grp 0    DP grp 1    DP grp 2    DP grp 3   ← different requests in each            r0: kv head 0 + q heads 0-7,  r1: kv head 1 + q heads 8-15  (same in every pair)MoE (EP = 8, one group spanning the whole server):           r0: E0-31     r1: E32-63    r2: E64-95    r3: E96-127           r4: E128-159  r5: E160-191  r6: E192-223  r7: E224-255Same GPU, two roles:  rank 5 = "TP rank 1 of DP group 2" for attention                             = "owner of experts 160-191" for MoE        └──────────────────────────────────────────────────────────────────┘
```

KV cache per token, across the 10 full-attention layers:

```
2 (K and V) x 2 KV heads x 256 dim x 2 bytes x 10 layers = 20 KB per token
KV stored per token (all 8 GPUs)    duplicationTP = 8                   80 KB                               4x   (each KV head on 4 GPUs)DP = 4 x TP = 2          20 KB                               1x   (each request lives in one pair)
```

That’s **about 4x more tokens in the KV cache**, which means more concurrent requests.

Weight memory per GPU:

```
TP = 8                    DP-attn + EP = 8experts                   32.2 B / 8 ≈ 4.0 B        32.2 B / 8 ≈ 4.0 B       ≈ 8 GBattention / other         ≈ 2.8 B / 8 ≈ 0.35 B      ≈ 2.8 B / 2 ≈ 1.4 B      (copied in each DP group)total weights             ≈ 8.7 GB                  ≈ 10.8 GB
```

The EP layout spends about 2 GB more per GPU on weights, because the attention weights are copied across the 4 DP groups. In exchange it saves 4x on the KV cache, which is usually the much larger number. (The 30 GDN layers also keep a fixed-size recurrent state per sequence. It’s split differently in the two layouts, but it’s similar in total.)

Suppose 64 requests are running, 16 in each DP group. Each request produces one token per step:

```
each DP group: its own batch of B_g tokens (1 per running request)     B_g = 16, B_total = 64   │   ▼ attention / GDN in the TP pair → all-reduce (2 GPUs)   │       each pair works only on its own 16 requests' KV cache   │   ▼ router: top-8 per token   │       16 tokens x 8 = 128 (token, expert) pairs per group   │   ▼ all-to-all DISPATCH across all 8 GPUs  (tokens from all 4 DP groups mix here)   │       each GPU sends ≈ 8 tokens x 5.3 GPUs x 4 KB ≈ 170 KB  → tiny, latency-bound   │   ▼ each GPU: grouped GEMM on its 32 experts   │       rows per expert ≈ (Σ B_g) · 8 / 256 = B_total / 32 = 2   │       experts actually hit per step ≈ 256 · (1 - (1 - 1/32)^64) ≈ 222 of 256   │   ▼ all-to-all COMBINE → tokens return to their DP group   │   ▼ weighted sum of 8 expert outputs + residual → next layer         (x 40 layers → 80 all-to-alls per generated token)
```

What decode tells us:

Now suppose each DP group is prefilling 4 prompts of 1,024 tokens, which is 4,096 tokens per group and 16,384 in total:

```
each DP group: 4 prompts x 1,024 tokens = 4,096 tokens   │   ▼ attention / GDN in the TP pair → all-reduce (2 GPUs)   │       4,096 tokens x 4 KB = 16 MB per all-reduce → bandwidth matters now   │   ▼ router: top-8 per token   │   ▼ all-to-all DISPATCH across all 8 GPUs   │       each GPU sends ≈ 2,048 tokens x 5.3 GPUs x 4 KB ≈ 42 MB per layer   │   ▼ each GPU: grouped GEMM on its 32 experts   │       rows per expert ≈ 16,384 x 8 / 256 = 512   │       gate_up: [512 x 2048] · [2048 x 1024]   ← a real, compute-bound GEMM   │       down:    [512 x 512]  · [512 x 2048]   │   ▼ all-to-all COMBINE → same ≈ 42 MB back
```

What prefill tells us:

```
                                TP = 8                      DP-attn (4x2) + EP = 8----------------------------    -------------------------   ---------------------------attention split                 2 q heads, KV copied 4x     8 q heads, 1 KV head, no copyKV cache per token (total)      80 KB                       20 KBexperts per GPU                 256 slices (1/8 each)       32 whole expertsexpert GEMM shape (gate_up)     2048 x 128                  2048 x 1024expert weight bytes per GPU     same                        samerows per expert (same batch)    same                        sameMoE collective                  all-reduce of all tokens    2 x all-to-all, routed onlyload balance                    perfect                     router-dependentweights per GPU                 ≈ 8.7 GB                    ≈ 10.8 GBmain win                        simplicity, balance         4x KV room → bigger batches,                                                            better expert GEMM shapes
```

If you need more throughput, there are two directions:

PP only enters the picture if a single replica can’t fit in one server, which isn’t the case for this model.

What I originally saw in the trace (every GPU running thin slices of all 256 experts) wasn’t wrong. It was simply TP = 8 applied everywhere. Once you see the four strategies as four different ways of splitting the work, the better layout for this model is clear.

[DP, TP, PP and EP: How LLM Parallelism Actually Fits Together](https://pub.towardsai.net/dp-tp-pp-and-ep-how-llm-parallelism-actually-fits-together-2cc4e611da72) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.
