Negative result: tail-latency gating for KV offload doesn't beat LRU+TTL (runnable simulator)

A runnable simulator published on GitHub as kv-cache-weigh-station v4.3 found that tail-quantile admission gating for HBM-to-host KV cache promotion did not beat plain LRU+TTL host retention with signal prefetch, producing −0.0% goodput with a 95% CI of [−2.3, +1.1]. Under predictive prefetch, a simple p_use > θ threshold matched the gated approach, and the author's next step is a single-GPU measurement of decode slowdown under concurrent KV DMA. The repository ships ./reproduce.sh, which runs in about 2 minutes on Mooncake traces, with the protocol and kill criteria in the release v4.3 PDF.

Runnable simulator for a KV offload negative result: GitHub - MrBeeboh/kv-cache-weigh-station: Negative result: tail-gated KV admission vs LRU+TTL host retention — protocol, simulator, kill criteria · GitHub https://github.com/MrBeeboh/kv-cache-weigh-station ./reproduce.sh , about 2 min, Mooncake traces . I tested tail-quantile admission gating for HBM↔host KV promotion against plain LRU+TTL host retention with signal prefetch. The gate did not win: −0.0% goodput, 95% CI −2.3, +1.1 . Under predictive prefetch, a simple p use θ threshold matched it. My next step is a single-GPU measurement of decode slowdown under concurrent KV DMA. If anyone has numbers for cudaMemcpyAsync versus gather-kernel copies during decode, I’d appreciate a pointer. The protocol and kill criteria are in the PDF release v4.3 : Release v4.3 · MrBeeboh/kv-cache-weigh-station · GitHub https://github.com/MrBeeboh/kv-cache-weigh-station/releases/tag/v4.3 Disclosure: AI assistants helped draft the paper and write the simulator; they are named in the paper.