Disaggregated inference is an inference optimization technique in which we run prefill and decode on separate GPU pools, and ship the KV cache between them over the networkFirst laid out in DistServe and Splitwise. The KV cache for a request is produced all at once during prefill, then consumed token by token during decode, so it's a natural place to cut the workload in two..
People usually think about it as a tool to tune time-per-output-token (TPOT) and time-to-first-token (TTFT) SLOs independently. It’s actually much stronger than that. I want to argue that, in practice, with sufficient load, and modulo many important implementation difficulties, you should always disaggregate.
The baseline that disaggregated prefill setups beat is either temporal disaggregation, or chunked prefill. Both run both prefill & decode on the same GPUs, but they schedule them differently.
Temporal disaggregation: we have some queue of waiting requests. We prefill some number of those requests. Then we run decode on the requests we’ve prefilled. When we want to top up our decode batch, we stop decoding, switch to prefill mode, do some prefilling, and then continue decoding, with our batch size now larger by however many requests we prefilled.
This works fine if we don’t really care whether users occasionally have to wait a long time between output tokens. But if we do care (we probably should), then it’s bad that these breaks we put in to prefill other users’ requests will inflate our all-important TPOT quantiles.
Chunked prefill is an improvement. The idea is our inference engine works on ‘heterogeneous batches’ — containing both decode requests and requests that prefill some number of tokens. Importantly: the number of tokens that get prefilled doesn’t have to be a whole sequence: we can build up the KV cache for a sequence across contiguous chunks.
The promise of the SARATHI paper that introduced the idea was that we would get clean throughput benefits: prefill is compute-heavy and memory-light, decode is memory-heavy and compute-light: if we put them together we get one of them for free. In practice, this doesn’t really work out: splitting a prefill into chunks means we have to transfer the antecedent KV cache times rather than just . You want large, so you don’t inflate TPOT, but a large amplifies that memory traffic, and inflates TTFT. But scheduling in a unified way across prefills and decodes, and being more granular in how we schedule prefill chunks meant that it was reasonably possible to hit SLOs in aggregated inference.
There are new visions for how to do aggregated inference better with various ways of partitioning the SMs on a GPU between the two rolesBullet splits the SMs between the phases with SM masks. DuetServe adapts the split each iteration. CUDA green contexts are the driver's supported mechanism for the partitioning..
The idea that everything is circulating around though is that these are really two different jobs, and it’s pretty sensible, when you have two different jobs, to split them out and run them independently. This is disaggregated prefill. In principle, the only cost to doing this disaggregation is that now the KV cache that the prefill generates and the decode needs in order to proceed has to be transferred from the prefill nodes to the decode nodes. In exchange, your SLOs don’t interfere with one another.
So, we break out our inference into two stages: prefill and decode. The decode stage consumes the KV cache that’s produced by the prefill stage, and the pipeline works properly so long as the consumption rate matches the production rate. How can we make sure they’re rate-matched?
Each request arrives with prompt tokens and will go on to generate output tokens (unknown a priori). Say that a GPU doing nothing but prefill gets through prompt tokens at a rate . The request then needs
GPU-seconds of prefill.
On the other hand, the decode phase produces one token per sequence per forward step, across its running batch of sequences. The GPU-time attributable to a single request’s output tokens is therefore . Over the generation of OSL output tokens the request needsWith for the balance set as the largest batch size that satisfies your TPOT SLO, or that fits into HBM, if you don't have one.
GPU-seconds of decode.
Now start pushing a request rate of requests per second through the system. In steady state, the prefill pool has to supply GPU-seconds of prefill work every second, and the decode pool GPU-seconds of decode.
It drops out that the balanced allocation of GPUs is prefillers and decoders, where:
is therefore a property of both the traffic mix (by which we mean ISL & OSL) and the rate at which prefill and decode pools process input tokens and produce output tokens (, , ).
The conditions under which we can disaggregate properly
The traffic mix changes over time. For a deployment to stay balanced, and for disaggregation to be always as good as aggregated deployments, three conditions have to hold.
- The total number of GPUs required to serve the traffic has to be high enough that the split rounds to whole workers.
- The fabric between the pools has to transport KV cache at the rate at which the prefillers produce it.
- The traffic balance has to be trackable: either the ISL/OSL mix is static, requests are routed so that different pools see roughly static mixes, or pools can be rebalanced faster than the mix moves.
When all three hold, a disaggregated deployment is never worse than an aggregated one.
The load vs. quantization effects
The main reason that we don’t see disaggregated prefill everywhere: it requires substantial load to beat the fact we can no longer use fractional amounts of GPUs to perform prefill or decode. One pool will be short by up to half a worker, so the loss is a fraction of order the worker width over the pool size. For single-GPU workers this falls off as . DeepSeek’s V3 deployment prefills in 32-GPU units and decodes in 320-GPU units, and rounding those takes thousands of GPUs.
The fabric sustains the KV cache production rate
This is more of a sanity check when scoping out whether your setup can support disaggregation: the flow of produced KV cache has to fit through the NICsOr the scale-up links. Every prompt token the prefillers compute sends bytes of cache to the decoders. Let and be the numbers of usable egress and ingress NICs, with bandwidths and , and let be the bandwidth available through the fabric itself. For prefill GPUs, the condition is
Prefill is compute-bound, so the cache production rate can be worked out from the chips FLOPs. Prefilling a single token for a model with active parameters costs FLOPs. If the chip has peak FLOPs , then we prefill tokens per secondThis doesn't account for cache hits. If you treat your prefill nodes as the sources of cache hits, and then transfer their whole cache downstream to the decoders, then you amplify the required bandwidth by a factor , which cache hit ratio, which makes this a much more stringent test. But you can also cache on the decoders, and arrange things such that you only transfer newly prefilled cache along the prefill-decode link, which we assume here..
As an example: GLM-5.2 stores 95 KB per token, and activates about 40B parameters. A B200 at its FP8 roofline of 4.5 PFLOP/s can prefill 56k GLM 5.2 tokens per second, which works out to 5.3 GB/s of generated KV cache. Its 400 Gb/s NICOne 400 Gb/s ConnectX-7 per GPU is the DGX B200 configuration. Other B200 systems pair different NICs. We assume the (impossible) 1xB200 for GLM5.2 situation because it safely bounds the more realistic deployments. can carry 50 GB/s, so the per-prefiller egress term is safely below the link bound.
The traffic balance can be tracked
The two pools are joined by the flow of KV cache between them. prefillers produce bytes of prompt cache per second. Write for the flow of prompt cache that admission takes in. The split is the choice that makes .
Between production and admission the cache sits in a queue, held in whatever memory the prefill pool has spareThe queue lives in spare prefiller HBM, plus host memory if it can keep up with the prefill transfer, which it probably can. The capacity is net of transfers in flight: the prefiller frees a request's blocks only once the decoder confirms receipt, so the transfer window holds its bytes on both ends. By Little's law the duplication comes to times the length of that window.. Write for the queue’s capacity and for the stock it holds.
Traffic mixes change! If the pool sizes are fixed, the flows no longer match. Write for the gap a persistent shift in the mix opens. Maybe the users that just woke up want more reasoning tokens than the ones that went to sleep, maybe everybody wants to send in PDFs this morning. Closing the gap means resizing the pools, but a new worker takes to come up — the engine cold start time. Until the autoscaling transition hits, the queue fills or drains at bytes per second.
A temporary surplus () fills the queue. While the headroom lasts, nothing is lost: the queued requests wait longer, but every GPU keeps doing useful work. When it runs out, prefill has to be backpressured. Decode carries on at full rate — output is decode-limited either way — but new requests pile up upstream and TTFT climbs.
A temporary shortfall () drains the stock. When it empties, decode batch slots sit emptyThis is the classical safety stock of inventory theory, held over a replenishment lead time. By Little's law a stock of bytes draining at bytes per second adds seconds of waiting to every request that passes through it. Building it is free only when there is prefill capacity to spare: during a surplus transient, or on the ramp into peak. In steady saturation, building the stock has to underfeed the decoders somewhere., and the decoders are more idle than they could be. Every request that passes through it waits on the users hot path, so its size is capped by the TTFT slack. The other defence is to let the decode workers run the missing prefills themselves (this is dynamo’s defence: conditional disaggregation). The cost is TPOT: all the drawbacks of chunked prefill that we discussed apply.
The deployment rides out the drift without stalling when the resize lands before the queue hits either end:
There are three ways to be more responsive to load: hold more capacity, start workers faster, or organize requests so that each pool sees a steadier mix and stays small.
Left of centre a shortfall drains the stock , right of centre a surplus fills the headroom . Below the boundaries and the resize lands before the queue hits an end and the shock is absorbed. Past them, the shading deepens with the average stranded flow over the window: decode output lost on the left, prefill backpressured on the right. The flow scale is the roofline figure from the fabric condition, 5.3 GB/s per prefill GPU.
Why it’s strictly better, once the conditions are met
So far disaggregation only ties: a balanced, well-fed disaggregated deployment matches the aggregated one. Scaling prefill workers and decode workers individually then gives us an extra degree of freedom we can use to set our SLO goals, should we choose to. What takes it a step further: there are many opportunities that come from specialisation.
Dynamic memory usage is lower. For aggregated inference, you have to allocate activation space for the largest operation you might perform on each GPU. That largest operation is usually a prefill. In disaggregated prefill, only the prefill nodes pay this, while the decoders spend the reclaimed space on batch.You can use different parallelisms. The appropriate degree of parallelism depends on the workload shape. One example: DeepSeekfor V3served prefill at EP32 and decode at EP320.You can tune the different phases separately. There are efficient ways to write kernels for prefill, and efficient ways to write kernels for decode. It is much harder to write a kernel that is efficient when you are prefilling, when you are decoding, and when you have a batch that mixes prefills and decodes.DeepEPships ahigh-throughput kernel and a low-latency kernelfor exactly this reason.Rank imbalances. With data-parallel attention (or any deployment where the EP domain is larger than the attention-parallel domain), many attention ranks feed into the same shared expert domain, and the dispatch is a barrier they all meet. If some ranks are working on prefills while others decode, every rank waits on the slowest, and the imbalance turns into GPU idle time.You can use different hardware. This one is underutilized, since most people buy their hardware off the shelf, and there aren’t any great widely available ‘decode-only’ chips yet.
Once properly tuned, disaggregated setups ought to always be stronger than aggregated ones.
If disaggregated prefill is so much better, why doesn’t everyone deploy disaggregated? Once you clear the fabric condition, what’s left is scale and headache. Either you don’t have the load for to round cleanly, or you don’t want to run the machinery that keeps the balance tracked.
The machinery is getting easier to run: dynamo does the balance-tracking for you, cold starts keep coming down, and the NICs between the pools keep getting faster. That leaves scale, and inference loads are growing into it.
Thanks to Charles Frye of Modal for useful feedback on a draft of this post.
A resource we haven’t really considered here, except insofar as we assume we need some extra on the prefill buffers: storage space. Will extra buffer space cost us throughput? I’ve seen a related misconception in papers in the field arguing for aggregated inference. It runs like this: A disaggregated deployment has twice as many GPUs, but only half of them hold KV cache for decoding. Aggregated inference on the same hardware has twice the KV capacity, so twice the running batch, so (since decode throughput scales with batch size) twice the throughput. Disaggregation wastes half your HBM. This is an accounting error, and unpicking it shows why there’s space to spare for KV cache buffering.
Let’s start with why we care about KV cache capacity. Decode throughput on a worker is the number of sequences in the running batch divided by the step time, , and the batch size is capped by how much KV cache fits in HBMThe reason is that the batch size amortizes the weight transfer. Weight transfer dominates the step time for small sequence length (where small depends on the number of KV bytes per token), so is roughly flat in and throughput is linear in it. For long sequences and sufficiently large KV cache, we can be dominated by KV cache transfer instead, in which case increasing the batch size doesn't increase throughput.. So more capacity means more throughput, and this is the sense in which twice the capacity is supposed to mean twice the throughput.
But capacity is only worth something while it’s decoding. The quantity that matters is the product of capacity and the fraction of time that capacity spends producing tokens.
Take the workload from the last section, with prefill fraction , and give the system GPUs.
Aggregated: every GPU carries a full-HBM batch , but spends of its time prefilling, and while it prefills its resident KV cache produces nothingChunked prefilling smears this out (prefill tokens ride along with decode steps instead of interrupting them) but doesn't change the accounting: the prefill FLOPs still come out of the same pool of GPU-seconds, and every decode step that shares a batch with a prefill chunk runs slower. Worse, chunked prefill requires you to do more memory movement since the KV cache must be transferred with every chunk.. Token output is .Disaggregated: give GPUs to prefill and to decode. The decode workers carry the same full-HBM batch and decode all the time: output .
In both cases, they’re the same number! Disaggregation doesn’t create idle HBM: it moves already idle HBM onto specific (prefill) GPUs.
Insofar as there is a tax on HBM, it’s that the KV cache has to be duplicated on the prefill and decode worker while a transfer is in flight. This comes from RDMA semantics: the prefiller can only free a request’s cache once the completion of the transfer is certain, and in the engines as shipped the unit of release is the whole request, not the block.
How long the window is depends on the engine. vLLM’s NIXL connector transfers after the whole prefill finishes: the decoder allocates blocks reads the cache across, and sends a notification to the prefiller to drop its cache. SGLang’s Mooncake path pushes chunk by chunk as the prefill proceeds, but the chunks stay resident on the prefiller’s radix cache until the request completes anyway. The reason it has to be this way is that the prefill engine usually does some kind of chunked prefill to bound the peak activation size (many other reasons too: HoL blocking, p95 TTFT, etc.). So you can’t get rid of earlier KV blocks as you create them, because you’ll need them again to prefill the next chunkOne nice thing about disagg prefill is it lets you think in extremes: on a prefill node, you can be almost arbitrarily compute-bound if you choose to be (though it has QoS impacts). This gives you space to do other crazy stuff 'for-free' while the compute unit is saturated: like pull KV cache chunks back over the NIC from the decoder (since that direction is undersaturated, you get the full bandwidth), or offload some KV cache to the host..
The same accounting applies to more granular disaggregation: pushing different parts of the model onto different accelerators. The most interesting example is attention-FFN disaggregation, where we put the attention operation on some set of devices, and the FFN operation on some other set.
We don’t want some set of accelerators idle while the others are working, so we need microbatches, then batch can be in attention while is in FFN, and then vice versa.
Again, what matters is both space, and utilization of that space. In aggregated deployment, the KV cache is utilized when we’re in the attention stage, and unutilized when the model is in the FFN stage. Just as in prefill-decode disaggregation, by disaggregating the two, we don’t waste space, we just put the existing wasted space onto specific acceleratorsThere doesn't need to be wasted space, since we could size the FFN group such that the FFN matrices + activations take up all the space. But we're much better off scaling the FFN group so its forward pass matches the forward pass of the attn. operation, so we don't get pipeline bubbles: the lack of this free parameter means the FFN devices end up with free HBM. An aggregated deployment would have had that memory doing work, so to keep the accounting even we should find it a use as well..
@misc{doubleword-when-to-disaggregate,
title = {The case for disaggregated LLM serving},
author = {Fergus Finn},
year = {2026},
howpublished = {Doubleword Blog},
url = {https://blog.doubleword.ai/when-to-disaggregate},
}