{"slug": "negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable", "title": "Negative result: tail-latency gating for KV offload doesn't beat LRU+TTL (runnable simulator)", "summary": "A runnable simulator published on GitHub as kv-cache-weigh-station v4.3 found that tail-quantile admission gating for HBM-to-host KV cache promotion did not beat plain LRU+TTL host retention with signal prefetch, producing −0.0% goodput with a 95% CI of [−2.3, +1.1]. Under predictive prefetch, a simple p_use > θ threshold matched the gated approach, and the author's next step is a single-GPU measurement of decode slowdown under concurrent KV DMA. The repository ships ./reproduce.sh, which runs in about 2 minutes on Mooncake traces, with the protocol and kill criteria in the release v4.3 PDF.", "body_md": "Runnable simulator for a KV offload negative result: [GitHub - MrBeeboh/kv-cache-weigh-station: Negative result: tail-gated KV admission vs LRU+TTL host retention — protocol, simulator, kill criteria · GitHub](https://github.com/MrBeeboh/kv-cache-weigh-station) (`./reproduce.sh`, about 2 min, Mooncake traces). I tested tail-quantile admission gating for HBM↔host KV promotion against plain LRU+TTL host retention with signal prefetch. The gate did not win: −0.0% goodput, 95% CI [−2.3, +1.1]. Under predictive prefetch, a simple p_use > θ threshold matched it. My next step is a single-GPU measurement of decode slowdown under concurrent KV DMA. If anyone has numbers for cudaMemcpyAsync versus gather-kernel copies during decode, I’d appreciate a pointer. The protocol and kill criteria are in the PDF (release v4.3): [Release v4.3 · MrBeeboh/kv-cache-weigh-station · GitHub](https://github.com/MrBeeboh/kv-cache-weigh-station/releases/tag/v4.3)\n\n*Disclosure: AI assistants helped draft the paper and write the simulator; they are named in the paper.*", "url": "https://wpnews.pro/news/negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable", "canonical_source": "https://discuss.huggingface.co/t/negative-result-tail-latency-gating-for-kv-offload-doesnt-beat-lru-ttl-runnable-simulator/190421#post_1", "published_at": "2026-10-12 18:20:14+00:00", "updated_at": "2026-10-12 18:49:14.424261+00:00", "lang": "en", "topics": ["ai-infrastructure", "machine-learning", "large-language-models"], "entities": ["GitHub", "MrBeeboh/kv-cache-weigh-station", "Mooncake", "HBM", "LRU+TTL", "cudaMemcpyAsync"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable", "markdown": "https://wpnews.pro/news/negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable.md", "text": "https://wpnews.pro/news/negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable.txt", "jsonld": "https://wpnews.pro/news/negative-result-tail-latency-gating-for-kv-offload-doesn-t-beat-lru-ttl-runnable.jsonld"}}