How Tokens Reach Their Experts During LLM Inference

DeepSeek-V4.1-Flash, a Mixture-of-Experts model with 384 experts per MoE module that routes each token to 6 of them, requires tokens to travel between GPUs in every one of its 40 layers when served with expert parallelism, according to an analysis of SGLang's source code measured on one machine with eight B300 GPUs. With 48 experts per GPU and 4,096-token prefill chunks, about 177 MB of tokens arrive at each GPU per layer when tokens are sent only to GPUs holding their experts, versus 320 MB if every token goes to every GPU; a decode step with 64 requests per GPU moves under 3 MB per round trip. SGLang's `--moe-a2a-backend` flag accepts twelve values for this all-to-all step, yet the SGLang cookbook recipe for DeepSeek-V4.1-Flash does not pass the flag at all.

Most of the large open models of the last two years are Mixture-of-Experts MoE models. In each layer, a small router looks at a token and picks a handful of experts for it, so only a fraction of the weights is used for any one token. DeepSeek-V4.1-Flash, for example, has 384 experts in every MoE module, and the router picks 6 of them per token. This works nicely on a single GPU. But once the model is spread over several GPUs with expert parallelism, each GPU holds only some of the experts, so a token is usually not on the GPU where its experts are. It has to travel there, and the result has to travel back, in every one of the 40 layers. Let’s say we serve this model on a machine with 8 GPUs and give each GPU 48 of the 384 experts. If every GPU is working on a prompt chunk of 4,096 tokens, then about 177 MB of tokens arrive at each GPU in every layer, provided that each token is only sent to the GPUs that hold one of its experts. If we simply send every token to every GPU, it is 320 MB. During decoding, a step with 64 requests per GPU moves less than 3 MB, but it still makes 40 of these round trips. So, how should the tokens be sent? In SGLang, the inference engine that I use as the reference in this series, the flag for this step, --moe-a2a-backend a2a is short for all-to-all , accepts twelve values. And the recipe that the SGLang cookbook recommends for this very model doesn’t pass the flag at all. Why are there so many ways to do this one step, what really differs between them, and when do we need one at all? In this article, I want to go through these ways one at a time, starting with the simplest one, and ask the same questions of each, which section 1.6 spells out. I will focus on what the GPUs send to each other and on what the receiving GPU has to do before its experts can run. I won’t cover the kernels that fuse the whole step with the expert computation DeepGEMM’s MegaMoE and its FlashInfer counterpart , since they get an article of their own. The mechanisms come from reading the source code, which I link along the way. The concrete numbers are for DeepSeek-V4.1-Flash https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash , and the measurements are my own, from one machine with eight B300 GPUs. 1. Why Tokens Have to Travel 1-why-tokens-have-to-travel If you have worked with expert parallelism before, you can skim this section. I would still read sections 1.3 and 1.4, since the rest of the article keeps coming back to two things from them: the number of copies of a token, and the problem that nobody knows in advance how many copies a GPU will receive. 1.1 Dispatch and Combine 11-dispatch-and-combine For every token, the router produces the ids of the chosen experts together with one weight per expert, and the output of the MoE module is the sum of the chosen experts’ outputs, each multiplied by its weight. With expert parallelism, each GPU holds a subset of the experts in full, which is 48 experts per GPU for DeepSeek-V4.1-Flash on 8 GPUs. A token then needs two trips. In the first one, called dispatch, its hidden state is sent to the GPUs that hold its experts. In the second one, called combine, the experts’ outputs come back to the GPU where the token started, and they are added up there. Figure 1 below follows a single token through both. Note in Figure 1 that the token needs three experts, but two of them E2 and E3 sit on the same GPU, so two copies of the token are enough. How large is one copy? The hidden state of DeepSeek-V4.1-Flash has 5,120 elements, and as 16-bit floats that’s 10,240 bytes, or 10 KB. I will call one copy of a token inside a tensor a row. I will call the kernel that runs all the experts of one GPU over their rows the expert kernel. And I will use two batch sizes as examples: a prefill batch, where the server processes a chunk of a prompt and each GPU has 4,096 rows to send, and a decode step, where each running request contributes one row and each GPU has 64. 1.2 Where the Tokens Are Before the MoE Module 12-where-the-tokens-are-before-the-moe-module Whether anything has to be sent at all depends on what happened in the attention module just before. If the attention module is split with tensor parallelism, the partial results of the GPUs are summed, and after that sum, every GPU holds the complete hidden state of every token. The alternative is called DP attention DP for data parallelism . Here, each GPU runs the complete attention module, but only for its own share of the requests, so it ends up holding only their tokens. Models with DeepSeek-style attention are often served this way because their KV cache would otherwise be stored on every GPU. Figure 2 below shows both situations. On the left side of Figure 2, every expert already has its tokens on its own GPU, whichever experts the router picks. On the right side, each token sits on one GPU, and most of its experts sit elsewhere. So the trips in Figure 1 are only needed in the second situation. When we ask SGLang for one of the all-to-all backends under tensor-parallel attention, it creates the second situation on purpose by letting each GPU keep only its slice of the batch. 1.3 How Many Copies 13-how-many-copies Now, to how many GPUs do we send a token? There are three natural answers, and they cost different amounts. The first is to send it to every GPU, which gives G copies on G GPUs. The second is to send one copy per expert, which gives 6 copies for our model, no matter how the experts are distributed. The third is the one from Figure 1: one copy for each GPU that holds at least one of the token’s experts. For the third answer, we can calculate the average. With E experts and k experts per token, a given GPU holds E/G experts, and the chance that none of the k chosen experts is among them is C E − E/G, k / C E, k , if the router picks uniformly. So, a token needs G × 1 − C E − E/G, k / C E, k GPUs on average. For DeepSeek-V4.1-Flash on 8 GPUs, this gives 4.43, so we have 8 copies, 6 copies, or 4.43 copies per token. All three numbers count the copy that stays on the token’s own GPU, which is one in eight on average. In bytes, a prefill batch with 4,096 rows per GPU delivers 320 MB, 240 MB, or 177 MB to each GPU in every layer. For a decode step with 64 rows per GPU, it’s 5.0 MB, 3.8 MB, or 2.8 MB. A real router doesn’t pick uniformly, so to see how far off the formula is, I recorded the router’s choices for a 7,700-token prompt in all 40 layers of the model. Figure 3 below shows the three answers for different numbers of GPUs, together with the recorded values. In Figure 3, the recorded values sit on the curve: with 8 GPUs, a token needed 4.41 GPUs on average, against the formula’s 4.43. Also, the gray line keeps growing with the number of GPUs, whereas the other two stay at or below 6. With 4 GPUs, sending a token everywhere costs 4 copies instead of 3.30. With 64 GPUs, it would cost 64 instead of 5.81. 1.4 Nobody Knows How Many Rows Will Arrive 14-nobody-knows-how-many-rows-will-arrive If every token goes to every GPU, each GPU knows that it will receive exactly G times the batch. As soon as the destination depends on the router, a GPU no longer knows how many rows it will receive. And the load is not even. In my recording, with 8 GPUs, the busiest GPU of a layer received on average 1.39 times as many rows as the average GPU, and in the worst layer 1.64 times. This is a practical problem, since the receiving side has to put the rows somewhere. Either the GPUs first tell each other how much they are going to send, and the receiver allocates a tensor of exactly the right size. Or the receiver reserves room for the worst case in advance, and a small counter next to the data says how many rows are valid. The first way is tidy, but it means that the CPU has to wait in the middle of the step, since it is the CPU that allocates tensors, and it can’t do so before it has read the count from the GPU. The wait itself is short, but it stops the CPU from queueing the following kernels ahead of the GPU. For a prefill batch, that hardly matters next to the work, but for decoding, it costs much more. A decode step is so small that the cost of launching its kernels from Python becomes noticeable, which is why inference engines record the sequence of launches once, with fixed shapes and fixed memory addresses, and replay it for every step. This is called a CUDA graph. A step in which the CPU reads a number from the GPU and then decides on the size of a tensor can’t be part of such a recording. So, the tidy way costs us a CPU round trip in each of the 40 layers, and the CUDA graph on top of it. 1.5 What the Expert Kernel Wants 15-what-the-expert-kernel-wants The receiving GPU also has to care about the order of the rows. The expert kernel multiplies each row by the weights of one particular expert, so it needs to know which rows belong to which expert. The kernels that we will meet do this in one of three ways, and Figure 4 below shows them for the same five tokens. From left to right in Figure 4, a contiguous kernel wants the rows of each expert next to each other, so a token that needs two experts appears twice. The contiguous kernel of DeepGEMM, DeepSeek’s kernel library, also wants each expert’s run padded to a multiple of 128 rows. A masked kernel gets a block of a fixed size for each expert, along with a count that tells it how many rows of the block to use. And a routed kernel takes the rows in whatever order they arrived, as long as each row comes with the ids of its experts. The first layout needs the exact count, so it belongs to the tidy way of the previous section. The other two have a fixed size, so they fit into a CUDA graph. 1.6 Summary 16-summary For each way of doing the round trip, I will ask the same three questions. How many copies of a token are sent? Does the CPU have to wait for a row count? And in what layout do the rows arrive, that is, what is left to do before the expert kernel can run? We will go through five answers: no backend at all, the two modes of the first version of DeepEP, DeepEP v2, and FlashInfer’s all-to-all. 2. No Backend: Every GPU Gets Every Token 2-no-backend-every-gpu-gets-every-token If we don’t pass --moe-a2a-backend at all, its value is none , and expert parallelism still works. So, what happens to the round trip? 2.1 Run Your Own Experts on Everything 21-run-your-own-experts-on-everything Nobody decides where a token should go. Every GPU gets every token, runs its own experts on the rows that need one of them, and skips the rest. Each GPU then holds a partial output for every token, which is zero where it had no expert to contribute. Adding up these partial outputs across the GPUs gives the output of the MoE module. With tensor-parallel attention, this needs no dispatch at all, because every GPU holds every token anyway. And the sum at the end is the same sum across the GPUs an all-reduce that a tensor-parallel MoE module would have needed. In this setup, SGLang’s dispatcher only translates the expert ids, so that the expert kernel can tell its own experts from the others simplified from here https://github.com/sgl-project/sglang/blob/f1065423/python/sglang/srt/layers/moe/token dispatcher/standard.py : Simplified from StandardDispatcher.dispatch local ids = torch.full num experts, , -1 -1: not on this GPU local ids first local : first local + num local = torch.arange num local topk ids = local ids topk ids the expert kernel skips -1 return hidden states nothing is sent This is exactly what the cookbook recipe for DeepSeek-V4.1 does. Its command line contains --tp 4 --ep-size 4 , which means tensor-parallel attention over 4 GPUs, with the experts divided among the same 4 GPUs. 2.2 With DP Attention: Gather and Reduce-Scatter 22-with-dp-attention-gather-and-reduce-scatter With DP attention, the tokens are not on every GPU, so they have to be brought there first. SGLang does this with the two collectives that the first article https://how-gpus-talk.vercel.app/ of this series described for a tensor-parallel MoE module: a gather before the experts, which gives every GPU the rows of all the GPUs, and a reduce-scatter after them, which adds up the partial outputs and returns to each GPU only the rows of its own tokens. Figure 5 below shows the whole pass. The faded rows in Figure 5 are the price of this approach, since the expert kernel of every GPU is handed all 8 rows. For a decode batch, the gather is usually an all-gather of rows that are padded to the same length on every GPU. When several GPUs are prefilling at once, SGLang instead lets every GPU write its rows at its own offset into a zero-filled buffer and sums the buffers with an all-reduce, which moves about twice as many bytes. 2.3 Why This Is Often Enough 23-why-this-is-often-enough So, why does the recipe for DeepSeek-V4.1 get away without a backend? The first reason is that it uses tensor-parallel attention, so there is nothing to dispatch. The second reason is the number of GPUs, and it would hold even with DP attention. On 4 GPUs, a gather delivers 4 copies of each token, whereas sending each token only where it is needed would deliver 3.30. That’s 21% more bytes, and in return, we need no extra buffers, no counting, and no library. On 8 GPUs, the gather delivers 8 copies instead of 4.43, which is 81% more. This is an argument about prefill. For a decode step, which moves a few megabytes, the number of kernels and handshakes per round trip matters more than the bytes. On our machine with 8 GPUs, a gather and a reduce-scatter for a decode step with 64 rows per GPU take {{agrs total 64 8}} microseconds together, and for a prefill batch with 4,096 rows per GPU, {{agrs total 4096 8}} milliseconds. We will compare these numbers with the alternatives in section 8. 2.4 Summary 24-summary To summarize in terms of our three questions, without a backend, a token is delivered to all G GPUs and under tensor-parallel attention, it is there already . The CPU never waits, because every GPU receives exactly G times the batch. And the rows arrive in the order of the senders with nothing left to do, but the expert kernel is handed all G times the batch and has to skip the rows it has no expert for. The obvious weakness is the number of copies, since with 8 GPUs, 3.6 of the 8 copies of a token go to GPUs that have no use for them. 3. DeepEP v1, Normal Mode: Exact Sizes, and the CPU Waits 3-deepep-v1-normal-mode-exact-sizes-and-the-cpu-waits The best-known library for sending tokens only where they are needed is DeepEP https://github.com/deepseek-ai/DeepEP , which DeepSeek open-sourced in February 2025. Its first version, which --moe-a2a-backend deepep selects in SGLang, has two modes that work quite differently. Let’s start with normal mode, which is the one for large batches. 3.1 Writing into Another GPU 31-writing-into-another-gpu All the remaining mechanisms in this article rely on one thing. Inside a machine with NVLink, a kernel that runs on one GPU can write directly into the memory of another GPU, as long as that memory was set up for it. Every GPU allocates a buffer at startup, and each buffer is mapped into the address space of all the other GPUs. After that, a kernel on GPU 0 can store a row into GPU 3’s buffer with an ordinary store instruction. This arrangement is called symmetric memory. Between machines, the network card does the same job, which is called RDMA. 3.2 Four Steps 32-four-steps With this in place, normal mode dispatches a batch in four steps. First, a small kernel goes through the expert ids of all the tokens and works out, for every token, the set of GPUs that hold at least one of its experts. This is where the 4.43 copies from section 1.3 come from: a token is sent once to each GPU in its set, however many of its experts live there. The same kernel counts how many rows each GPU will get. Second, the GPUs exchange these counts by writing them into each other’s buffers, and each GPU adds up how many rows it is about to receive. Third, the CPU waits for this number. The following excerpt is shortened from DeepEP’s dispatch function https://github.com/deepseek-ai/DeepEP/blob/567632d/csrc/deep ep.cpp L598-L656 : // Simplified from Buffer::intranode dispatch moe recv counter = -1; // memory that the CPU and the GPU share notify dispatch ... ; // a GPU kernel exchanges the counts while true { // the CPU spins here num recv tokens = static cast<int moe recv counter ; if num recv tokens = 0 break; } auto recv x = torch::empty {num recv tokens, hidden}, x.options ; Fourth, the rows move. Half of the kernel’s thread blocks act as senders and the other half as receivers. Each sender writes its rows into a queue that lives in the receiver’s memory, a chunk at a time, and the receiver takes them out and puts them into the output tensor. Figure 6 below shows the whole sequence, with the CPU in its own lane. In Figure 6, the dispatch is two rounds of GPU work with a CPU step in between, and the last box, the regrouping, is done by SGLang, as the next section explains. 3.3 What Arrives 33-what-arrives What the receiving GPU gets is a tensor with exactly as many rows as were sent to it, in the order in which they arrived, plus the expert ids and weights of each row. SGLang pairs this mode with a contiguous kernel from DeepSeek’s DeepGEMM library , so the rows first have to be regrouped: SGLang copies each row once for each of its experts on this GPU, into one run of rows per expert. In terms of Figure 4, the 5 rows that arrived become the 7 rows on the left. 3.4 The Price 34-the-price The price of the exact size is the wait in the third step. SGLang turns off the CUDA graph altogether when this mode is forced for every batch, and the post in which the SGLang team described their first large deployment with DeepEP https://www.lmsys.org/blog/2025-05-05-large-scale-ep/ names the same two costs, the blocked CPU and the shapes that a CUDA graph can’t record. There is a second, smaller price. The transfer kernel occupies SMs, the compute units of a GPU, that the experts can’t use at the same time, and the number of SMs and the size of the chunks are not derived from anything. They come from a table in DeepEP’s source code with one hand-tuned entry per number of GPUs, and SGLang lets us override them with a JSON file. How long does all of this take? For a prefill batch with 4,096 rows per GPU on our 8 GPUs, a dispatch takes {{normal dispatch 4096 8}} milliseconds and a combine {{normal combine 4096 8}}. For 64 rows per GPU, it’s {{normal dispatch 64 8}} and {{normal combine 64 8}} microseconds, and during the dispatch, the CPU is blocked for {{normal host 64 8}} microseconds. 3.5 Summary 35-summary To summarize, normal mode sends one copy of a token to each GPU that needs it, which is 4.43 copies for our example. The CPU waits for the count in every layer. And the rows arrive in arrival order with their expert ids, after which SGLang copies them into one row per expert for the contiguous kernel. For prefill, this is a good trade, since 40 short waits don’t matter much next to the work of a large batch. A decode step, on the other hand, would pay for 40 CPU round trips and lose its CUDA graph. That’s why DeepEP has a second mode. 4. DeepEP v1, Low-Latency Mode: A Fixed Region per Expert 4-deepep-v1-low-latency-mode-a-fixed-region-per-expert How can the receiver do without the count? The second mode of DeepEP v1 answers this by deciding in advance where every row will go. 4.1 One Region per Expert and Sender 41-one-region-per-expert-and-sender In low-latency mode, the receive buffer of each GPU is divided into regions, with one region for every pair of a local expert and a sending GPU. Each region has room for the largest batch that a sender is allowed to have, which I will call the cap. It is a number of rows per GPU that is fixed at startup. Now, a sender doesn’t have to ask anyone where to write. For a token that needs expert E3 on GPU 1, GPU 0 simply takes the next free slot in the region “E3, from GPU 0” on GPU 1 and writes the row there. Since no two senders share a region, they can’t get in each other’s way, and since each region holds a full cap, it can’t overflow. The catch is that the regions are organized by expert, so a token is written once for each of its experts, even if two of them live on the same GPU. Figure 7 below shows what this looks like for the five tokens from Figure 4. In Figure 7, token a crosses twice, once for E2 and once for E3. For our example model, this means 6 copies per token instead of 4.43, or 35% more data. 4.2 The Count Stays on the GPU 42-the-count-stays-on-the-gpu The receiver still has to find out how many rows are in each region. For this, each sender writes the number of rows it has sent as the very last thing, after all of its rows. The receiving kernel waits until these numbers have arrived from all the senders, which also tells it that the rows are complete. This number never goes to the CPU. The receiving kernel packs the rows of each expert together and writes the count per expert into a small tensor on the GPU, and the output tensor always has the same shape, namely the number of local experts times the cap times the number of GPUs. In terms of Figure 4, this is the layout in the middle, and the expert kernel that reads it is a masked one. For our decode step with a cap of 128, that’s 48 × 8 × 128 = 49,152 rows of room for the 384 rows that arrive on average. Since nothing depends on a number that the CPU has to read, the whole step can be recorded in a CUDA graph. This mode was built for clusters where the tokens travel between machines via RDMA, so its send half only hands the rows to the network card, and the engine can run something else before it calls the receive half. Within one machine, the rows are written directly over NVLink. 4.3 The Price 43-the-price The buffer has to hold a full cap for every pair of an expert and a sender. For DeepSeek-V4.1-Flash on 8 GPUs, DeepEP asks for {{ll buffer 128}} GB per GPU with a cap of 128 rows and for {{ll buffer 1024}} GB with a cap of 1,024 rows, and the output tensor comes on top of that. The cap itself is limited to 1,024 rows per GPU by the way the kernel signals completion, and SGLang’s default is 128. If a batch is larger than the cap, the call fails, which is why prefill normally avoids this mode. SGLang’s default setting for DeepEP is therefore a mixture: batches that contain prompt tokens anywhere go through normal mode, and pure decode batches go through low-latency mode. On our machine, a decode step with 64 rows per GPU takes {{ll dispatch 64 8}} microseconds for the dispatch and {{ll combine 64 8}} for the combine. For comparison, the DeepEP v1 README reports 77 and 114 microseconds for 128 tokens per GPU on 8 H800 GPUs, although that measurement sent the rows via RDMA, with 7,168 elements per row and 8 experts per token. 4.4 Summary 44-summary To summarize, low-latency mode sends one copy per expert, which is 6 copies for our example. The CPU doesn’t wait. And the rows arrive grouped by expert in blocks of a fixed size, ready for a masked expert kernel. Let’s put the two modes next to each other. Normal mode sends few copies and allocates exactly, and it pays with a CPU wait. Low-latency mode has no wait, and it pays with more copies and a buffer for the worst case. An engine that wants both ends up with two layouts, two paths into the expert kernel, and a rule for switching between them. Can a single path avoid both costs? 5. DeepEP v2: Send Once per GPU, Lay Out Afterwards 5-deepep-v2-send-once-per-gpu-lay-out-afterwards In April 2026, DeepSeek released the second version of DeepEP, a rewrite around a single class called ElasticBuffer . In SGLang, it is a separate backend, deepep v2 . So, how does one path serve both prefill and decoding? 5.1 One Transfer, Two Switches 51-one-transfer-two-switches If we look at where the two modes of v1 differ, it comes down to two things: whether the CPU waits for the count, and whether a token is copied once per GPU or once per expert. V2 makes the first one an argument, and it moves the second one to the receiving GPU. A dispatch in v2 always sends a token once to each GPU that needs it, as normal mode did. Each receiving GPU keeps a region for every sender, so the rows can be written straight into place, as in low-latency mode. The GPUs still exchange their counts, since the receiver needs them to know where the rows of each expert will begin, but the counts stay on the GPUs unless the caller asks for them with do cpu sync . The layout by expert is then built by a second kernel that DeepEP calls the copy epilogue. It runs on the receiving GPU, right after the first kernel, and copies the rows from the regions into the output tensor. If the caller sets do expand , it writes one row for each pair of a token and a local expert, grouped by expert. Figure 8 below shows the two stages for our five tokens. If we compare Figure 8 with Figure 7, the result on the right is almost the same: token a ends up in the rows of E2 and in the rows of E3. The difference is that the second copy crossed NVLink in Figure 7, whereas in Figure 8 it is made inside the memory of one GPU. Without the CPU wait, the output tensor can’t have an exact size, so v2 allocates it for the worst case, and a short list of offsets on the GPU tells the expert kernel where the rows of each expert begin and end. V2 also does away with the tuning table from section 3.4. The number of SMs for the transfer is computed from a small model of the hardware: the NVLink bandwidth of the GPU, how many bytes per second one SM can read and write, and the expected number of copies per token from section 1.3. For our model on 8 B300 GPUs, this formula asks for 26 SMs. SGLang doesn’t run anything else during the transfer, so it lets DeepEP take at least 64 SMs of the B300’s 148. 5.2 What SGLang Does with It 52-what-sglang-does-with-it How does SGLang use the two switches? For a decode batch, it asks for no CPU wait and for the layout by expert, so the dispatch fits into the CUDA graph. For a batch with prompt tokens, it asks for the CPU wait, as in normal mode. For decoding, SGLang’s masked kernel expects the blocks of a fixed size per expert from section 4. So after DeepEP has grouped the rows by expert, SGLang copies them once more, into such blocks. As far as I can tell, this second copy only exists because the kernel was written for the layout of v1. For prefill, DeepEP’s layout by expert is currently only used on Hopper GPUs. On Blackwell GPUs such as our B300, SGLang still takes the rows in arrival order and does the regrouping itself, as in section 3.3. The pull request that turned this layout on for Hopper https://github.com/sgl-project/sglang/pull/37261 measured what it is worth there: on two Hopper machines, the prompt throughput of a MiMo model went from 25,256 tokens per second with DeepEP v1 to 27,029 with v2 and to 29,107 with v2’s layout by expert, for 8,000-token prompts. What does this add up to in a server? The pull request that added the backend https://github.com/sgl-project/sglang/pull/35634 compared it with v1 for DeepSeek-V4-Flash on 8 B200 GPUs. Decoding was 2.8% slower 8,429 against 8,675 tokens per second , and prefill was 2.9% faster 32,447 against 31,533 . So the two versions are close on a single machine. 5.3 From NVSHMEM to NCCL, and V2.5 53-from-nvshmem-to-nccl-and-v25 DeepEP v1 used a separate NVIDIA library, NVSHMEM, to set up symmetric memory and to let the GPU issue RDMA writes. V2 gets both from NCCL, the library that PyTorch already uses for its collectives. More precisely, it uses a recent device-side interface of NCCL that kernels can call directly, which the README calls the “NCCL Gin backend”. The flip side is that DeepEP now depends on the exact NCCL version that sits under PyTorch’s own collectives. At the end of September 2026, DeepSeek released V2.5. It deletes the v1 code and with it the dependency on NVSHMEM, and it splits ElasticBuffer into a buffer for expert parallelism, now called EPBuffer , and three buffers for other purposes. It also requires NCCL 2.32.3 or newer. SGLang has not made this step yet. The build that I used ships DeepEP 2.1.0, which still contains both the v1 and the v2 class, along with NCCL 2.30.7, and the move to V2.5 is an open item on the roadmap for DeepSeek-V4.1 https://github.com/sgl-project/sglang/issues/42170 . 5.4 Summary 54-summary To summarize, DeepEP v2 sends one copy per GPU that needs the token, 4.43 for our example. The CPU waits only if the caller asks for it, which SGLang does for prefill. And the rows arrive in arrival order or, if requested, grouped by expert, after which SGLang copies them once more for its masked kernel during decoding, or regroups them itself for prefill on Blackwell. On our 8 GPUs, a dispatch and a combine with 64 rows per GPU take {{v2masked total 64 8}} microseconds in the decode setting, and with 4,096 rows per GPU, {{v2sync total 4096 8}} milliseconds in the prefill setting. 6. FlashInfer A2A: Rows with Their Routing 6-flashinfer-a2a-rows-with-their-routing The last of the five ways comes from NVIDIA. Its kernel was written for TensorRT-LLM, where it is known as the NVLink one-sided all-to-all, and it was ported to FlashInfer https://github.com/flashinfer-ai/flashinfer , the kernel library that SGLang builds on, at the end of 2025. In SGLang, it is selected with --moe-a2a-backend flashinfer . What if the receiving GPU doesn’t group anything? 6.1 A Block of Rows per Sender 61-a-block-of-rows-per-sender In this design, every GPU has a workspace in symmetric memory with one block of rows for each sender, and each block has room for the largest batch that any GPU has in this step. As in DeepEP v2, a token is sent once to each GPU that needs it. The kernel starts one thread block per token. Its first few threads look up the GPUs of the token’s 6 experts and agree on one thread per distinct GPU, so that two experts on the same GPU lead to a single copy see the kernel https://github.com/flashinfer-ai/flashinfer/blob/26f2418/csrc/nv internal/tensorrt llm/kernels/communicationKernels/moeAlltoAllKernels.cu . The row index at the destination comes from a counter, and the row is then stored directly into the destination’s workspace. What is new is that, next to the hidden state, the sender also writes the token’s expert ids and its weights. SGLang hands the kernel a list of three payloads for this shortened from its dispatcher https://github.com/sgl-project/sglang/blob/f1065423/python/sglang/srt/layers/moe/token dispatcher/flashinfer.py : Simplified from FlashinferDispatcher.dispatch payloads = hidden states, topk ids, topk weights recv = moe a2a.dispatch topk ids, payloads, runtime max tokens per rank, ... every tensor in recv has the shape num gpus, runtime max tokens per rank, ... Since the routing arrives with each row, the receiving GPU doesn’t have to sort anything. It passes the workspace to a routed expert kernel, which reads the ids and does the selection itself, as in the right panel of Figure 4. Figure 9 below shows the workspace of GPU 1 for our five tokens. In Figure 9, the five rows sit in the blocks of their senders, each with its expert ids and weights, and one slot stays empty. 6.2 Flags Instead of a Count 62-flags-instead-of-a-count How does GPU 1 know that all the rows have arrived? There is no count for the CPU here either. When a GPU has written its last row, it sets a completion flag in the workspace of each of the other GPUs, and then it waits until all of their flags have shown up in its own workspace. These are the check marks in Figure 9. This wait happens inside the kernel, on the GPU, so the shapes that the CPU sees never depend on it, and the step can be recorded in a CUDA graph. 6.3 The Price 63-the-price The price of the fixed shape here is that the expert kernel is handed a tensor with G times the largest batch as its number of rows, whatever arrived. In Figure 9, which leaves out GPU 1’s own block, that’s 6 rows for 5 tokens, and the empty slot is marked with an invalid expert id so that the kernel skips it. For our prefill example, it is 8 × 4,096 = 32,768 rows, which is as many as with no backend at all when every GPU has a full batch. So, compared with the gather from section 2, FlashInfer A2A saves bytes on the wire, but not rows in the expert kernel. The workspace has to hold these rows for both directions, which comes to {{fi workspace 4096}} GB per GPU for our model at 4,096 rows. Is it worth it? The pull request that added the backend https://github.com/sgl-project/sglang/pull/14668 reports a comparison on 4 GB200 GPUs with a 4-bit DeepSeek-R1 checkpoint and 2,048 concurrent requests: 19,421 output tokens per second with no backend whose gather already sent 4-bit rows , 18,459 with FlashInfer A2A sending 16-bit rows, and 19,826 with FlashInfer A2A sending 4-bit rows. In other words, on 4 GPUs and against a gather with the same wire format, the all-to-all won by about 2%. A more recent pull request https://github.com/sgl-project/sglang/pull/39818 switched prefill from the gather to FlashInfer A2A and measured GLM-5.2 on 4 B300 GPUs. With 4,096 prompt tokens per GPU, the prompt throughput rose by 6.1%. With 2,048 or fewer, nothing changed, since those batches already used the all-to-all. On our 8 GPUs, a dispatch and a combine take {{fi total 64 8}} microseconds with 64 rows per GPU and {{fi total 4096 8}} milliseconds with 4,096. 6.4 Summary 64-summary To summarize, FlashInfer A2A sends one copy per GPU that needs the token, which is 4.43 for our example. The CPU doesn’t wait, since the GPUs signal each other with flags inside the kernel. And the rows arrive in no particular order together with their routing and go to a routed expert kernel as they are, padding included. 7. Combine, and the Wire Format of Dispatch 7-combine-and-the-wire-format-of-dispatch The two halves of the round trip look symmetric in Figure 1, but they carry different things. What dispatch has to carry is decided by what the expert kernel accepts as input, and what combine has to carry is decided by what it produces. 7.1 Combine: Where the Sum Is Taken 71-combine-where-the-sum-is-taken The output of a token is a weighted sum over its 6 experts. We can split this sum into two stages: a sum over the experts that sit on the same GPU, and a sum over the GPUs. The five ways differ in which of the two stages happens before the rows travel, as Figure 10 below shows for the token from Figure 1. Let’s go through Figure 10 from left to right. Without a backend, every GPU multiplies and adds for its own experts, and the collective at the end of the layer adds up a full-width tensor from every GPU. With DeepEP v1 in normal mode and with DeepEP v2, the first stage also happens on the expert’s GPU, so only one row per GPU travels back, and the token’s home GPU adds up the rows it receives. In normal mode, it is SGLang that multiplies and pre-adds before it calls combine. With the layout by expert of v2, SGLang only multiplies, and DeepEP’s combine kernel does the pre-adding. Since low-latency mode sent one copy per expert, it also returns one row per expert, and the home GPU multiplies each row by its weight and adds them up. With FlashInfer A2A, the routed expert kernel already produces the weighted sum per GPU, and the combine call places it in that GPU’s workspace. The home GPU then reads its rows out of the other GPUs’ workspaces and adds them up in 32-bit floats. To summarize, each way returns as many rows as it sent: 8 per token without a backend, 6 in low-latency mode, and 4.43 in the others. 7.2 Dispatch: Fewer Bytes per Token 72-dispatch-fewer-bytes-per-token For dispatch, there is a different way to save. Quantized checkpoints have expert kernels that convert their input to 8-bit or 4-bit numbers before multiplying anyway. If the kernel accepts input that is already quantized, we can convert the token before sending it and send fewer bytes. For DeepSeek-V4.1-Flash, one copy of a token then shrinks as follows: 16-bit 10,240 bytes ████████████████████ FP8 5,280 bytes ██████████ NVFP4 2,880 bytes ██████ The FP8 row includes a one-byte scale factor for every 32 values, and the NVFP4 row one for every 16. DeepEP v1 uses a four-byte scale for every 128 values, which comes to the same size. All the backends offer some version of this. DeepEP v1 sends FP8 rows by default, DeepEP v2 follows the format of the expert weights, FlashInfer A2A sends NVFP4 or MXFP8 rows when the expert kernel takes that format as input and 16-bit rows otherwise, and even the gather from section 2 has a variant that quantizes to NVFP4 first. The GB200 post https://www.lmsys.org/blog/2025-09-25-gb200-part-2/ of the SGLang team describes the step from FP8 to NVFP4 in DeepEP as “halving the required network traffic”. Combine, in contrast, is 16-bit almost everywhere. What travels back is a sum of expert outputs that still has to be added to other rows, and nothing downstream wants it quantized. One exception is FlashInfer’s kernel, which can produce a quantized result for the next layer. 8. What It Costs on Eight B300 GPUs 8-what-it-costs-on-eight-b300-gpus So far, each section reported its timings by itself. Let’s put them next to each other. 8.1 How I Measured 81-how-i-measured I timed dispatch and combine by themselves, outside of a server, with one process per GPU and the calls copied from SGLang’s dispatchers. The rows are real: I served DeepSeek-V4.1-Flash with the cookbook flags under a load of ShareGPT conversations, recorded the hidden states that enter the MoE module together with the router’s expert ids and weights, and replayed them. Every GPU sends the same number of rows, and all rows are 16-bit unless noted. Each phase is timed with two CUDA events around it, which is the time a replayed CUDA graph would spend, and I report the median of 30 calls, repeated three times. A call that does nothing takes about 10 microseconds for dispatch and combine together in this setup, so that’s the floor. What is not included is everything outside the two calls: the expert kernel, quantization, and other kernels competing for SMs. Two entries of the comparison needed extra work. DeepEP V2.5 requires a newer NCCL than SGLang ships, so I built it in a separate environment. And FlashInfer’s main branch has a newer all-to-all kernel, called Cake, that SGLang doesn’t select yet, so I ran that from a source checkout. Since my machine has no RDMA network card, DeepEP v2 and V2.5 also needed EP DISABLE GIN=1 to start at all. 8.2 Dispatch and Combine 82-dispatch-and-combine Figure 11 below shows the time for a dispatch and a combine together on 8 GPUs, for decode sizes on the left and prefill sizes on the right. {{fig11 discussion 8}} 8.3 What Is Left to Do 83-what-is-left-to-do The third question of this article, what is left to do before the expert kernel can run, has a price as well, and I timed it for the three paths where SGLang has to copy rows. {{layout discussion}} 8.4 In a Server 84-in-a-server How much of a decode step is this? With the cookbook recipe 4 GPUs, tensor-parallel attention, no backend and 64 concurrent ShareGPT conversations, the server produced 4,615 output tokens per second with 12.2 ms between the tokens of a request. A round trip of {{best total 64 4}} microseconds in each of the 40 layers would come to {{best step share}} of such a step. I would have liked to swap the backends in that server and measure the difference directly. This is not possible yet: at the commit I used, a DeepSeek-V4.1 checkpoint with a vision encoder, such as V4.1-Flash, refuses to start with any all-to-all backend “V4.1 vision currently supports TP/EP/DP without CP, PP or MoE A2A” , and its roadmap lists DP attention, MegaMoE and DeepEP V2.5 as open items. For other models, the pull requests that I quoted in sections 5 and 6 found end-to-end differences of a few percent between the backends on a single machine. 8.5 Summary 85-summary {{s8 summary}} 9. The Flags in SGLang, and What People Deploy 9-the-flags-in-sglang-and-what-people-deploy 9.1 Twelve Values 91-twelve-values Let’s start by sorting the twelve values of --moe-a2a-backend from the introduction: - none is the default from section 2. - deepep is DeepEP v1 from sections 3 and 4, and --deepep-mode chooses between normal , low latency , and auto . - deepep v2 is section 5, and flashinfer is section 6. - mooncake , nixl , pplx , and mori are four more libraries, which I will get to in a moment. - megamoe and flashinfer megamoe select the fused kernels that I am leaving for another article. - ascend fuseep and ascend tp are for Huawei’s Ascend NPUs. Every value except none and ascend tp forces the experts to be divided among all the GPUs of the server. In SGLang’s terms, --ep-size is set equal to --tp-size , which is the total number of GPUs. 9.2 Setting the Cap 92-setting-the-cap Every way of avoiding the CPU wait reserves room for the worst case, which shows up in SGLang as one environment variable per backend that sets the cap from section 4.1, in rows per GPU. For DeepEP v1, the variable is SGLANG DEEPEP NUM MAX DISPATCH TOKENS PER RANK . It only applies to low-latency mode, its default is 128, and it can’t exceed 1,024. For DeepEP v2, it is SGLANG DEEPEP V2 NUM MAX DISPATCH TOKENS PER RANK , also with a default of 128, and it has to cover the prefill chunk as well. For FlashInfer A2A, it is SGLANG FLASHINFER NUM MAX DISPATCH TOKENS PER RANK , which defaults to the prefill chunk size or 4,096, whichever is larger. Note that the unit is rows, not requests. With speculative decoding, a request contributes several rows to a decode step, so 64 requests with 4 draft tokens each already need 256 rows. DeepEP v2 and FlashInfer A2A check the cap against the prefill chunk and the largest decode batch when the server starts. With DeepEP v1, we only find out when the first batch that is too large arrives. Below is the command that the cookbook gives for DeepSeek-V4-Flash on 4 B300 GPUs in its balanced setting, with DP attention and DeepEP v1: SGLANG DEEPEP NUM MAX DISPATCH TOKENS PER RANK=1024 \ sglang serve \ --model-path deepseek-ai/DeepSeek-V4-Flash-0731 \ --tp 4 --attn-dp-size 4 \ --moe-a2a-backend deepep \ --deepep-config '{"normal dispatch":{"num sms":96},"normal combine":{"num sms":96}}' \ --trust-remote-code The two newer backends come with more conditions. DeepEP v2 is only accepted for model architectures that have been registered for it, and only together with the DeepGEMM expert kernel. Its --deepep-v2-mode flag has the values direct and hybrid , which are not the two modes of v1, since direct is for a single machine with NVLink, and hybrid is for several machines, where a row first travels between the machines via RDMA and is then forwarded over NVLink. FlashInfer A2A requires DP attention in which every GPU has its own share of the requests --attn-dp-size equal to --tp , and its wire format from section 7.2 is chosen with --flashinfer-a2a-dispatch-type . 9.3 Four More Libraries 93-four-more-libraries What about mooncake , nixl , pplx , and mori ? None of them runs on my machine, so this section only relies on the source code and on their authors’ posts. Three of the four plug into SGLang through the interface of DeepEP’s low-latency mode, with a fixed region per expert and a count that stays on the GPU, and change the transport underneath. Mooncake https://github.com/kvcache-ai/Mooncake and NVIDIA’s NIXL https://github.com/ai-dynamo/nixl use this to survive the loss of a GPU, which DeepEP doesn’t offer. The fixed layout helps here: if a GPU stops sending, its regions simply stay empty, so all the receiver needs is a timeout and a list of the GPUs that are still alive. When that list changes, SGLang moves the experts of the missing GPU to the remaining ones and runs the step again. According to the post about this feature https://www.lmsys.org/blog/2026-03-25-eep-partial-failure-tolerance/ , a DeepSeek-V3.2 test on 32 GPUs was interrupted for less than 10 seconds after some of its processes were killed. pplx wraps the kernels that Perplexity published https://github.com/perplexityai/pplx-kernels for Hopper GPUs, which the upstream repository has meanwhile deprecated in favor of a successor. mori is AMD’s library for its own GPUs. It is the exception to the rule above, since SGLang runs it in the style of normal mode by default. It can also quantize in both directions: according to the post about it https://www.lmsys.org/blog/2026-05-28-mori/ , sending 4-bit rows and returning 8-bit rows reduces the bytes of a round trip by a factor of 2.56. 9.4 What the Recipes Choose 94-what-the-recipes-choose So, which of these do people deploy? Looking through the SGLang cookbook, the choice mostly follows from two things: where the tokens are before the MoE module, and how many GPUs share the experts. - For DeepSeek-V4.1, every recipe uses tensor-parallel attention on 4 or 8 GPUs and no backend. - For DeepSeek-V4-Flash on Blackwell GPUs, the low-latency recipe uses no expert parallelism at all. The balanced recipe uses DP attention with DeepEP v1, as in the command above, and the high-throughput recipe uses MegaMoE. - For GLM-5.2, the FP8 recipes for the balanced and high-throughput settings use DP attention with DeepEP v1. - For Qwen3.8 on four GB300 machines with 16 GPUs, the balanced and high-throughput FP8 recipes use DeepEP v2 in hybrid mode, and the high-throughput NVFP4 recipe uses FlashInfer A2A with a routed kernel. The open InferenceX benchmark has examples with many more GPUs. Its SGLang recipes for DeepSeek-R1 on GB200 racks decode with DeepEP in low-latency mode and spread the experts over up to 48 GPUs, and its vLLM recipes for DeepSeek-V4 select the FlashInfer kernel from section 6. Its SGLang recipes for DeepSeek-V4 used to select DeepEP as well, until a change https://github.com/SemiAnalysisAI/InferenceX/blob/57063af55c75d6b6ee8e2d159d47d94940ca0c05/inferencex-e2e/perf-changelog.yaml L3056 switched them to MegaMoE. 10. What I Left Out 10-what-i-left-out To keep this article at a manageable length, I skipped a few things: - MegaMoE in DeepGEMM and in FlashInfer, which fuse the round trip with the expert computation. - Expert load balancing, which moves or duplicates experts so that the GPUs receive similar numbers of rows. - Overlapping the round trip with other work, such as the shared expert or a second batch. - The backends for Ascend NPUs, and DeepEP’s buffers for purposes other than expert parallelism. - Measurements across several machines, since my one machine has no RDMA network card. 11. Summary 11-summary To wrap up, let’s collect the answers to the three questions from section 1.6 in one place. Figure 12 below shows them for the five ways of doing the round trip. {{summary text}}