A runnable simulator published on GitHub as kv-cache-weigh-station v4.3 found that tail-quantile admission gating for HBM-to-host KV cache promotion did not beat plain LRU+TTL host retention with sign…
Mixture of Experts has changed the structure of AI inference serving by altering which tensors are active per token, what must remain close together, which transfers need strong local bandwidth, and h…
A former B2B SaaS web developer who switched to inference engineering about a month ago published a skills map arguing the field requires only three service categories: routing and scheduling via Nvid…
A prefix-cache simulator replaying 68,266 requests from 393 real Claude Code sessions and 23,608 Mooncake requests failed to beat the production LRU baseline in three separate attempts, according to t…
VLLM's Decode Context Parallelism (DCP) enables higher concurrency and throughput for long-context agentic workloads by splitting the KV cache across GPUs, according to a new blog post from the vLLM t…
Prefill/decode disaggregation, which runs LLM prefill and decode phases on separate hardware pools, cut p99 inter-token latency by 66% (from 88ms to 30ms) in a Go simulation, at the cost of higher tim…
A new arXiv paper (arXiv:2607.28633v1) proposes a topology-aware transfer orchestrator for disaggregated GPU inference, claiming existing systems like DistServe, Splitwise, and Mooncake ignore that ba…
An engineer at vLLM analyzed concurrent KV block load coalescing in Mooncake and vLLM, finding that while the code path for single-flight optimization exists, real traces show almost no coalescable wo…
NVIDIA released DynoSim, a workload-driven discrete-event simulation of the Dynamo serving stack, to model the complex interactions between LLM serving components. The Rust-based simulator runs 1,500x…