{"slug": "the-case-for-disaggregated-llm-serving", "title": "The case for disaggregated LLM serving", "summary": "Disaggregated LLM serving, which runs prefill and decode on separate GPU pools and transfers KV caches over the network, should always be used in practice under sufficient load, according to a technical argument citing DistServe and Splitwise. The technique independently tunes time-to-first-token (TTFT) and time-per-output-token (TPOT) SLOs, avoiding interference between the two phases, at the cost of network transfer of KV caches. Alternatives like temporal disaggregation and chunked prefill (SARATHI) have drawbacks, while new approaches like Bullet and DuetServe partition GPU SMs within aggregated inference.", "body_md": "# The case for disaggregated LLM serving\n\nDisaggregated inference is an inference optimization technique in which we run\nprefill and decode on separate GPU pools, and ship the KV cache between them\nover the networkFirst laid out in [DistServe](https://arxiv.org/abs/2401.09670) and\n[Splitwise](https://arxiv.org/abs/2311.18677). The KV cache for a request is\nproduced all at once during prefill, then consumed token by token during\ndecode, so it's a natural place to cut the workload in two..\n\nPeople usually think about it as a tool to tune time-per-output-token (TPOT) and time-to-first-token (TTFT) SLOs independently. It’s actually much stronger than that. I want to argue that, in practice, with sufficient load, and modulo many important implementation difficulties, you should always disaggregate.\n\n[What is disaggregated prefill](#what-is-disaggregated-prefill)\n\nThe baseline that disaggregated prefill setups beat is either *temporal\ndisaggregation*, or *chunked prefill*. Both run both prefill & decode on the\nsame GPUs, but they schedule them differently.\n\nTemporal disaggregation: we have some queue of waiting requests. We prefill some number of those requests. Then we run decode on the requests we’ve prefilled. When we want to top up our decode batch, we stop decoding, switch to prefill mode, do some prefilling, and then continue decoding, with our batch size now larger by however many requests we prefilled.\n\nThis works fine if we don’t really care whether users occasionally have to wait a long time between output tokens. But if we do care (we probably should), then it’s bad that these breaks we put in to prefill other users’ requests will inflate our all-important TPOT quantiles.\n\nChunked prefill is an improvement. The idea is our inference engine works on ‘heterogeneous batches’ — containing both decode requests and requests that prefill some number of tokens. Importantly: the number of tokens that get prefilled doesn’t have to be a whole sequence: we can build up the KV cache for a sequence across contiguous chunks.\n\nThe promise of the [SARATHI paper](https://arxiv.org/abs/2308.16369) that\nintroduced the idea was that we would get clean throughput benefits: prefill is\ncompute-heavy and memory-light, decode is memory-heavy and compute-light: if we\nput them together we get one of them for free. In practice, this doesn’t really\nwork out: splitting a prefill into chunks means we have to transfer the\nantecedent KV cache times rather than just . You want large, so you\ndon’t inflate TPOT, but a large amplifies that memory traffic, and\ninflates TTFT. But scheduling in a unified way across prefills and\ndecodes, and being more granular in how we schedule prefill chunks meant that\nit was reasonably possible to hit SLOs in aggregated inference.\n\nThere are new visions for how to do aggregated inference better with various\nways of partitioning the SMs on a GPU between the two roles[Bullet](https://arxiv.org/abs/2504.19516) splits the SMs between the\nphases with SM masks. [DuetServe](https://arxiv.org/abs/2511.04791) adapts\nthe split each iteration. [CUDA green\ncontexts](https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__GREEN__CONTEXTS.html)\nare the driver's supported mechanism for the partitioning..\n\nThe idea that everything is circulating around though is that these are really\ntwo different jobs, and it’s pretty sensible, when you have two different jobs,\nto split them out and run them independently. This is *disaggregated prefill*.\nIn principle, the only cost to doing this disaggregation is that now the KV\ncache that the prefill generates and the decode needs in order to proceed has\nto be *transferred* from the prefill nodes to the decode nodes. In exchange,\nyour SLOs don’t interfere with one another.\n\n[How to rate balance](#how-to-rate-balance)\n\nSo, we break out our inference into two stages: prefill and decode. The decode stage consumes the KV cache that’s produced by the prefill stage, and the pipeline works properly so long as the consumption rate matches the production rate. How can we make sure they’re rate-matched?\n\nEach request arrives with prompt tokens and will go on to\ngenerate output tokens (unknown *a priori*). Say that a GPU\ndoing nothing but prefill gets through prompt tokens at a rate . The\nrequest then needs\n\nGPU-seconds of prefill.\n\nOn the other hand, the decode phase produces one token per sequence per forward step, across its running batch of sequences. The GPU-time attributable to a single request’s output tokens is therefore . Over the generation of OSL output tokens the request needsWith for the balance set as the largest batch size that satisfies your TPOT SLO, or that fits into HBM, if you don't have one.\n\nGPU-seconds of decode.\n\nNow start pushing a request rate of requests per second through the system. In steady state, the prefill pool has to supply GPU-seconds of prefill work every second, and the decode pool GPU-seconds of decode.\n\nIt drops out that the balanced allocation of GPUs is prefillers and decoders, where:\n\nis therefore a property of both the *traffic mix* (by which we mean ISL\n& OSL) and the rate at which prefill and decode pools process input tokens and\nproduce output tokens (, , ).\n\n[The conditions under which we can disaggregate properly](#the-conditions-under-which-we-can-disaggregate-properly)\n\nThe traffic mix changes over time. For a deployment to stay balanced, and for disaggregation to be always as good as aggregated deployments, three conditions have to hold.\n\n- The total number of GPUs required to serve the traffic has to be high enough that the split rounds to whole workers.\n- The fabric between the pools has to transport KV cache at the rate at which the prefillers produce it.\n- The traffic balance has to be trackable: either the ISL/OSL mix is static, requests are routed so that different pools see roughly static mixes, or pools can be rebalanced faster than the mix moves.\n\nWhen all three hold, a disaggregated deployment is never worse than an aggregated one.\n\n[The load vs. quantization effects](#the-load-vs-quantization-effects)\n\nThe main reason that we don’t see disaggregated prefill everywhere: it requires\nsubstantial load to beat the fact we can no longer use fractional amounts of\nGPUs to perform prefill or decode. One pool will be short by up to half a worker, so the\nloss is a fraction of order the worker width over the pool size. For single-GPU\nworkers this falls off as . DeepSeek’s [V3\ndeployment](https://arxiv.org/abs/2412.19437) prefills in 32-GPU units and\ndecodes in 320-GPU units, and rounding those takes thousands of\nGPUs.\n\n[The fabric sustains the KV cache production rate](#the-fabric-sustains-the-kv-cache-production-rate)\n\nThis is more of a sanity check when scoping out whether your setup can support disaggregation: the flow of produced KV cache has to fit through the NICsOr the scale-up links. Every prompt token the prefillers compute sends bytes of cache to the decoders. Let and be the numbers of usable egress and ingress NICs, with bandwidths and , and let be the bandwidth available through the fabric itself. For prefill GPUs, the condition is\n\nPrefill is compute-bound, so the cache production rate can be worked out from the chips FLOPs. Prefilling a single token for a model with active parameters costs FLOPs. If the chip has peak FLOPs , then we prefill tokens per secondThis doesn't account for cache hits. If you treat your prefill nodes as the sources of cache hits, and then transfer their whole cache downstream to the decoders, then you amplify the required bandwidth by a factor , which cache hit ratio, which makes this a much more stringent test. But you can also cache on the decoders, and arrange things such that you only transfer newly prefilled cache along the prefill-decode link, which we assume here..\n\nAs an example: [GLM-5.2](https://huggingface.co/zai-org/GLM-5.2/blob/main/config.json) stores\n95 KB per token, and activates about 40B parameters. A\n[B200](https://www.nvidia.com/en-us/data-center/dgx-b200/) at its FP8\nroofline of 4.5 PFLOP/s can prefill 56k GLM 5.2 tokens per second, which\nworks out to 5.3 GB/s of generated KV cache. Its\n400 Gb/s NICOne 400 Gb/s ConnectX-7 per GPU is the DGX B200 configuration. Other B200\nsystems pair different NICs. We assume the (impossible) 1xB200 for GLM5.2\nsituation because it safely bounds the more realistic deployments. can carry 50 GB/s, so the per-prefiller egress term is safely\nbelow the link bound.\n\n[The traffic balance can be tracked](#the-traffic-balance-can-be-tracked)\n\nThe two pools are joined by the flow of KV cache between them. prefillers produce bytes of prompt cache per second. Write for the flow of prompt cache that admission takes in. The split is the choice that makes .\n\nBetween production and admission the cache sits in a queue, held in whatever\nmemory the prefill pool has spareThe queue lives in spare prefiller HBM, plus host memory if it can keep\nup with the prefill transfer, which it probably can. The capacity is net of\ntransfers in flight: the prefiller frees a request's blocks only once the\ndecoder confirms receipt, so the transfer window holds its bytes on both ends.\nBy [Little's law](https://en.wikipedia.org/wiki/Little%27s_law) the duplication\ncomes to times the length of that window.. Write for the queue’s capacity and\nfor the stock it holds.\n\nTraffic mixes change! If the pool sizes are fixed, the flows no longer match.\nWrite for the gap a persistent shift in the mix opens. Maybe\nthe users that just woke up want more reasoning tokens than the ones that went\nto sleep, maybe everybody wants to send in PDFs this morning. Closing the gap\nmeans resizing the pools, but a new worker takes to come up — the\n[engine cold start time](https://fergusfinn.com/blog/fast-sglang-starts/). Until the autoscaling\ntransition hits, the queue fills or drains at bytes per second.\n\nA temporary surplus () fills the queue. While the headroom lasts, nothing is lost: the queued requests wait longer, but every GPU keeps doing useful work. When it runs out, prefill has to be backpressured. Decode carries on at full rate — output is decode-limited either way — but new requests pile up upstream and TTFT climbs.\n\nA temporary shortfall () drains the stock. When it empties, decode\nbatch slots sit emptyThis is the classical [safety\nstock](https://en.wikipedia.org/wiki/Safety_stock) of [inventory\ntheory](https://en.wikipedia.org/wiki/Inventory_theory), held over a\n[replenishment lead time](https://en.wikipedia.org/wiki/Lead_time). By\n[Little's law](https://en.wikipedia.org/wiki/Little%27s_law) a stock of\nbytes draining at bytes per second adds seconds of waiting to every\nrequest that passes through it. Building it is free only when there is prefill\ncapacity to spare: during a surplus transient, or on the ramp into peak. In\nsteady saturation, building the stock has to underfeed the decoders somewhere., and the decoders are more idle than they could be.\nEvery request that passes through it waits on the users hot path, so its size\nis capped by the TTFT slack. The other defence is to let the decode workers run\nthe missing prefills themselves (this is\n[dynamo](https://github.com/ai-dynamo/dynamo)’s defence: conditional\ndisaggregation). The cost is TPOT: all the drawbacks of chunked prefill that we\ndiscussed apply.\n\nThe deployment rides out the drift without stalling when the resize lands before the queue hits either end:\n\nThere are three ways to be more responsive to load: hold more capacity, start workers faster, or organize requests so that each pool sees a steadier mix and stays small.\n\nLeft of centre a shortfall drains the stock , right of centre a surplus fills the headroom . Below the boundaries and the resize lands before the queue hits an end and the shock is absorbed. Past them, the shading deepens with the average stranded flow over the window: decode output lost on the left, prefill backpressured on the right. The flow scale is the roofline figure from the fabric condition, 5.3 GB/s per prefill GPU.\n\n[Why it’s strictly better, once the conditions are met](#why-its-strictly-better-once-the-conditions-are-met)\n\nSo far disaggregation only ties: a balanced, well-fed disaggregated deployment matches the aggregated one. Scaling prefill workers and decode workers individually then gives us an extra degree of freedom we can use to set our SLO goals, should we choose to. What takes it a step further: there are many opportunities that come from specialisation.\n\n**Dynamic memory usage is lower**. For aggregated inference, you have to allocate activation space for the largest operation you might perform on each GPU. That largest operation is usually a prefill. In disaggregated prefill, only the prefill nodes pay this, while the decoders spend the reclaimed space on batch.**You can use different parallelisms**. The appropriate degree of parallelism depends on the workload shape. One example: DeepSeek[for V3](https://arxiv.org/abs/2412.19437)served prefill at EP32 and decode at EP320.**You can tune the different phases separately**. There are efficient ways to write kernels for prefill, and efficient ways to write kernels for decode. It is much harder to write a kernel that is efficient when you are prefilling, when you are decoding, and when you have a batch that mixes prefills and decodes.[DeepEP](https://github.com/deepseek-ai/DeepEP)ships a[high-throughput kernel and a low-latency kernel](https://fergusfinn.com/blog/anatomy-of-a-high-performance-ep-kernel/)for exactly this reason.**Rank imbalances**. With data-parallel attention (or any deployment where the EP domain is larger than the attention-parallel domain), many attention ranks feed into the same shared expert domain, and the dispatch is a barrier they all meet. If some ranks are working on prefills while others decode, every rank waits on the slowest, and the imbalance turns into GPU idle time.**You can use different hardware**. This one is underutilized, since most people buy their hardware off the shelf, and there aren’t any great widely available ‘decode-only’ chips yet.\n\nOnce properly tuned, disaggregated setups ought to always be stronger than aggregated ones.\n\n[Conclusion](#conclusion)\n\nIf disaggregated prefill is so much better, why doesn’t everyone deploy disaggregated? Once you clear the fabric condition, what’s left is scale and headache. Either you don’t have the load for to round cleanly, or you don’t want to run the machinery that keeps the balance tracked.\n\nThe machinery is getting easier to run:\n[dynamo](https://github.com/ai-dynamo/dynamo) does the balance-tracking for\nyou, [cold starts keep coming down](https://fergusfinn.com/blog/fast-sglang-starts/), and the NICs\nbetween the pools keep getting faster. That leaves scale, and inference loads\nare growing into it.\n\n*Thanks to Charles Frye of\nModal for useful feedback on a draft of this post.*\n\n[Appendix: Space & Utilization](#appendix-space--utilization)\n\nA resource we haven’t really considered here, except insofar as we assume we need some extra on the prefill buffers: storage space. Will extra buffer space cost us throughput? I’ve seen a related misconception in papers in the field arguing for aggregated inference. It runs like this: A disaggregated deployment has twice as many GPUs, but only half of them hold KV cache for decoding. Aggregated inference on the same hardware has twice the KV capacity, so twice the running batch, so (since decode throughput scales with batch size) twice the throughput. Disaggregation wastes half your HBM. This is an accounting error, and unpicking it shows why there’s space to spare for KV cache buffering.\n\nLet’s start with why we care about KV cache capacity. Decode throughput on a worker is the number of sequences in the running batch divided by the step time, , and the batch size is capped by how much KV cache fits in HBMThe reason is that the batch size amortizes the weight transfer. Weight transfer dominates the step time for small sequence length (where small depends on the number of KV bytes per token), so is roughly flat in and throughput is linear in it. For long sequences and sufficiently large KV cache, we can be dominated by KV cache transfer instead, in which case increasing the batch size doesn't increase throughput.. So more capacity means more throughput, and this is the sense in which twice the capacity is supposed to mean twice the throughput.\n\nBut capacity is only worth something while it’s decoding. The quantity that matters is the product of capacity and the fraction of time that capacity spends producing tokens.\n\nTake the workload from the last section, with prefill fraction , and give the system GPUs.\n\n**Aggregated**: every GPU carries a full-HBM batch , but spends of its time prefilling, and while it prefills its resident KV cache produces nothingChunked prefilling smears this out (prefill tokens ride along with decode steps instead of interrupting them) but doesn't change the accounting: the prefill FLOPs still come out of the same pool of GPU-seconds, and every decode step that shares a batch with a prefill chunk runs slower. Worse, chunked prefill requires you to do more memory movement since the KV cache must be transferred with every chunk.. Token output is .**Disaggregated**: give GPUs to prefill and to decode. The decode workers carry the same full-HBM batch and decode all the time: output .\n\nIn both cases, they’re the same number! Disaggregation doesn’t create idle HBM: it moves already idle HBM onto specific (prefill) GPUs.\n\n[The HBM tax that does exist](#the-hbm-tax-that-does-exist)\n\nInsofar as there is a tax on HBM, it’s that the KV cache has to be duplicated on the prefill and decode worker while a transfer is in flight. This comes from RDMA semantics: the prefiller can only free a request’s cache once the completion of the transfer is certain, and in the engines as shipped the unit of release is the whole request, not the block.\n\nHow long the window is depends on the engine. vLLM’s\n[NIXL](https://github.com/ai-dynamo/nixl) connector transfers after the whole\nprefill finishes: the decoder allocates blocks reads the cache across, and\nsends a notification to the prefiller to drop its cache. SGLang’s\n[Mooncake](https://github.com/kvcache-ai/Mooncake) path pushes chunk by chunk\nas the prefill proceeds, but the chunks stay resident on the prefiller’s radix\ncache until the request completes anyway. The reason it has to be this way is\nthat the prefill engine usually does some kind of chunked prefill to bound the\npeak activation size (many other reasons too: HoL blocking, p95 TTFT, etc.). So\nyou can’t get rid of earlier KV blocks as you create them, because you’ll need\nthem again to prefill the next chunkOne nice thing about disagg prefill is it lets you think in extremes: on\na prefill node, you can be almost arbitrarily compute-bound if you choose to be\n(though it has QoS impacts). This gives you space to do other crazy stuff\n'for-free' while the compute unit is saturated: like pull KV cache chunks back\nover the NIC from the decoder (since that direction is undersaturated, you get\nthe full bandwidth), or offload some KV cache to the host..\n\n[Attention-FFN disaggregation](#attention-ffn-disaggregation)\n\nThe same accounting applies to more granular disaggregation: pushing different parts of the model onto different accelerators. The most interesting example is attention-FFN disaggregation, where we put the attention operation on some set of devices, and the FFN operation on some other set.\n\nWe don’t want some set of accelerators idle while the others are working, so we need microbatches, then batch can be in attention while is in FFN, and then vice versa.\n\nAgain, what matters is both space, and utilization of that space. In aggregated deployment, the KV cache is utilized when we’re in the attention stage, and unutilized when the model is in the FFN stage. Just as in prefill-decode disaggregation, by disaggregating the two, we don’t waste space, we just put the existing wasted space onto specific acceleratorsThere doesn't need to be wasted space, since we could size the FFN group such that the FFN matrices + activations take up all the space. But we're much better off scaling the FFN group so its forward pass matches the forward pass of the attn. operation, so we don't get pipeline bubbles: the lack of this free parameter means the FFN devices end up with free HBM. An aggregated deployment would have had that memory doing work, so to keep the accounting even we should find it a use as well..\n\n```\n@misc{doubleword-when-to-disaggregate,\n  title        = {The case for disaggregated LLM serving},\n  author       = {Fergus Finn},\n  year         = {2026},\n  howpublished = {Doubleword Blog},\n  url          = {https://blog.doubleword.ai/when-to-disaggregate},\n}\n```\n\n", "url": "https://wpnews.pro/news/the-case-for-disaggregated-llm-serving", "canonical_source": "https://blog.doubleword.ai/when-to-disaggregate", "published_at": "2026-08-11 09:00:00+00:00", "updated_at": "2026-08-11 09:43:55.914715+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research"], "entities": ["DistServe", "Splitwise", "SARATHI", "Bullet", "DuetServe", "NVIDIA"], "alternates": {"html": "https://wpnews.pro/news/the-case-for-disaggregated-llm-serving", "markdown": "https://wpnews.pro/news/the-case-for-disaggregated-llm-serving.md", "text": "https://wpnews.pro/news/the-case-for-disaggregated-llm-serving.txt", "jsonld": "https://wpnews.pro/news/the-case-for-disaggregated-llm-serving.jsonld"}}