{"slug": "to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving", "title": "To Keep or Not to Keep: Learning KV Cache Retention in Disaggregated LLM Serving Systems", "summary": "Researchers released KVLearn, a learning-based framework for KV-cache retention in disaggregated LLM serving, reporting end-to-end time-to-first-token reductions of up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines, with inter-node KV transfer volume cut by up to 53% versus LRU-Pool. KVLearn combines a Prefix Reuse Predictor that estimates reuse probability from structural and temporal prefix features without modifying model weights, a Cost-Aware Retention Score that converts reuse probability into keep/admit decisions using recomputation, transfer and storage costs, and an Adaptive Threshold Controller that adjusts admission thresholds online from observed hit rates and memory pressure. On the MM-Session workload, throughput stays within about 5% of oracle, and the code is available at github.com/FastLM/KVLearn.", "body_md": "arXiv:2610.28637v1 Announce Type: new \nAbstract: Disaggregated \\textsc{LLM} serving separates prefill and decode across node pools, placing a network fabric between \\textsc{KV}-cache computation and consumption. This breaks a key assumption of classical cache policies: a miss no longer incurs only local recomputation, but also network transfer to the decode node. Likewise, admitting a block to the global cache requires an additional transfer, so poor retention decisions waste both memory and bandwidth. We present \\textbf{KVLearn}, a learning-based framework for cost-aware \\textsc{KV}-cache retention in disaggregated serving. KVLearn includes: (i) a lightweight Prefix Reuse Predictor (\\textsc{PRP}) that estimates reuse probability from structural and temporal prefix features without modifying model weights; (ii) a Cost-Aware Retention Score (\\textsc{CARS}) that converts reuse probability into keep/admit decisions using recomputation, transfer, and storage costs; and (iii) an Adaptive Threshold Controller (\\textsc{ATC}) that adjusts admission thresholds online from observed hit rates and memory pressure. We integrate KVLearn into a globally disaggregated serving topology and evaluate it on text and multimodal workloads, including large image/video-derived \\textsc{KV} blocks with heterogeneous reuse patterns. KVLearn reduces end-to-end time-to-first-token by up to 56% versus No-Cache, 38% versus LRU-Pool, and 33% versus Mooncake-style baselines, while cutting inter-node \\textsc{KV} transfer volume by up to 53% versus LRU-Pool. On MM-Session, throughput remains within ${\\sim}5%$ of oracle. Code is available at https://github.com/FastLM/KVLearn.", "url": "https://wpnews.pro/news/to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving", "canonical_source": "https://arxiv.org/abs/2610.28637", "published_at": "2026-10-22 04:00:00+00:00", "updated_at": "2026-10-22 04:13:26.236239+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "machine-learning", "ai-research"], "entities": ["KVLearn", "Prefix Reuse Predictor", "Cost-Aware Retention Score", "Adaptive Threshold Controller", "Mooncake", "FastLM"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving", "markdown": "https://wpnews.pro/news/to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving.md", "text": "https://wpnews.pro/news/to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving.txt", "jsonld": "https://wpnews.pro/news/to-keep-or-not-to-keep-learning-kv-cache-retention-in-disaggregated-llm-serving.jsonld"}}