# One Long Prompt Shouldn't Freeze Everyone's Tokens: Prefill/Decode Disaggregation > Source: > Published: 2026-08-05 16:21:43+00:00 *An LLM request is two workloads in a trench coat — a heavy, bursty prefill and a stream of tiny latency-sensitive decodes. Running them on the same engines lets one big prompt stall everyone. Splitting them fixes it.* **TL;DR:** Every LLM request is two very different jobs. **Prefill** reads the whole prompt — heavy, bursty, and slow for long contexts. **Decode** then emits tokens one at a time — tiny, but *latency-sensitive*. Run them on the same engines and a big prefill jumps ahead of everyone's decodes: **head-of-line blocking**, and the token stream stutters. **Prefill/decode disaggregation** puts prefill and decode on separate pools so decodes never queue behind a prefill. In a runnable Go simulation, splitting the pools cut **p99 inter-token latency by 66%** (88ms → 30ms) — trading a little time-to-first-token for a far smoother stream. This is now standard practice in frontier serving stacks ([DistServe](https://arxiv.org/abs/2401.09670), [Splitwise](https://arxiv.org/abs/2311.18677), [vLLM × Mooncake](https://vllm.ai/blog/2026-05-06-mooncake-store)). **Mental model:** a coffee shop with one worker who both grinds beans and pours espresso. A customer orders a giant batch grind and everyone waiting for a simple pour is stuck behind it. Split the shop into a *grinder station* and a *pour station* and the pours keep flowing no matter how big the grind. Serving an LLM token is not one kind of work — it's two: Colocate them — the default — and they fight for the same compute. Continuous batching helps throughput but doesn't remove the conflict: when the engine runs a long prefill, the decodes it's also hosting have to wait. And they can't flee to a less-busy engine, because *this* engine holds their KV cache. So one long-context prompt lands and every active token stream on that engine stutters. Your p99 ITL is hostage to your longest prompt. Disaggregation ([DistServe](https://arxiv.org/abs/2401.09670), [Splitwise](https://arxiv.org/abs/2311.18677)) separates the two phases onto different hardware: Now a giant prefill can't block anyone's decodes, because it physically runs on a different pool. Each pool can also be tuned and scaled independently for its own SLO (TTFT for prefill, ITL for decode) instead of compromising on one knob for both. The simulation pins each request's decodes to the engine that ran its prefill (KV-cache locality) — the constraint that makes colocated blocking unavoidable: ``` if disaggregated { if j.prefill { server, dur = 0, w.prefill[j.req] // prefill pool } else { server, dur = 1, decodeDur // decode pool — never behind a prefill } } else { server = j.req % 2 // pinned engine holds this request's KV cache // ...its decodes are stuck behind whatever prefill lands here } Prefill/Decode Disaggregation — keep long prefills from stalling the token stream before → after: p99 inter-token latency 88ms (colocated) → 30ms (disaggregated) (66% lower) 240 requests, 2 servers, 20% long-context bursts (700–1800ms prefill), 20 decode steps × 5ms. layout p99 token lat mean token lat mean TTFT colocated (shared) 88 ms 15 ms 426 ms disaggregated (split) 30 ms 11 ms 1032 ms ``` Same workload, same total hardware (2 engines each). Colocated lets bursty prefills jump ahead of tiny decodes, so the p99 token stutters to 88ms. Disaggregated isolates decodes and holds p99 to 30ms — a **66% smoother** stream. The honest cost is **TTFT**: with only one engine dedicated to prefill, first tokens arrive later (426ms → 1032ms). That's the real knob disaggregation gives you — pool sizing lets you buy back TTFT by provisioning prefill and decode independently. **Reality check:** these are simulated queue latencies, not GPU numbers — directional only. The direction is well-established: DistServe and Splitwise report multiples-higher goodput under latency SLOs from PD disaggregation, and the real-world win hinges on how cheap your KV-cache transfer is and how much prefill actually contends with decode. Disaggregation went from research idea to default in two years. [DistServe](https://arxiv.org/abs/2401.09670) showed that separating prefill and decode and sizing each for its own SLO can serve **multiples more requests** under latency constraints; [Splitwise](https://arxiv.org/abs/2311.18677) made the same case for splitting the phases across different hardware. By 2026 it's productized: vLLM ships PD disaggregation, and the [vLLM × Mooncake](https://vllm.ai/blog/2026-05-06-mooncake-store) work pairs it with a distributed KV cache pool so prefill and decode engines — even on different machines — share caches over fast transport. The load-bearing enabler is exactly the KV-cache handoff this demo hand-waves. The transferable idea generalizes past LLMs: **when one queue mixes bursty heavy work with steady latency-sensitive work, isolate them.** It's the same instinct as separating batch from interactive traffic, or OLAP from OLTP — applied at the token level. It models queueing and head-of-line blocking, not GPUs: "prefill" and "decode" are service times, and it ignores continuous batching, the real (non-zero) cost of KV-cache transfer, and memory pressure — all of which real systems must handle, and which is why cheap KV transport matters so much. Two honest caveats: disaggregation isn't free (you pay a transfer and a TTFT hop, visible above), and it only pays off when prefill bursts actually contend with latency-sensitive decodes. Short, uniform prompts under light load won't show the gap — measure your ITL tail before splitting. ``` go run . # standard library only ``` **Papers** **Engineering**