Prefill/decode disaggregation, which runs LLM prefill and decode phases on separate hardware pools, cut p99 inter-token latency by 66% (from 88ms to 30ms) in a Go simulation, at the cost of higher tim…
A new arXiv paper (arXiv:2607.28633v1) proposes a topology-aware transfer orchestrator for disaggregated GPU inference, claiming existing systems like DistServe, Splitwise, and Mooncake ignore that ba…
An engineer at vLLM analyzed concurrent KV block load coalescing in Mooncake and vLLM, finding that while the code path for single-flight optimization exists, real traces show almost no coalescable wo…
NVIDIA released DynoSim, a workload-driven discrete-event simulation of the Dynamo serving stack, to model the complex interactions between LLM serving components. The Rust-based simulator runs 1,500x…