{"slug": "varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms", "title": "VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs", "summary": "VarRate, a training-free variable-rate KV cache compression method for long-context LLMs, outperforms token-selection methods by allocating low-rank budgets based on query salience instead of evicting tokens, degrading by only 3.5-5.5 points where query-aware selection collapses by 11-15 points. At a matched 20% budget on LongBench, VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B, and matches KVzip in accuracy with one-eighth the prefill overhead.", "body_md": "arXiv:2607.15498v1 Announce Type: new\nAbstract: The key-value (KV) cache is the main memory bottleneck in long-context large language model (LLM) inference. Two leading training-free families are both structurally limited: token-selection methods (SnapKV, Ada-KV) score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget. We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none. Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench (16 tasks), VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.", "url": "https://wpnews.pro/news/varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms", "canonical_source": "https://arxiv.org/abs/2607.15498", "published_at": "2026-07-20 04:00:00+00:00", "updated_at": "2026-07-20 13:40:57.577168+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["VarRate", "SnapKV", "Ada-KV", "KVzip", "Llama-3.1-8B", "Qwen2.5-7B", "LongBench"], "alternates": {"html": "https://wpnews.pro/news/varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms", "markdown": "https://wpnews.pro/news/varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms.md", "text": "https://wpnews.pro/news/varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms.txt", "jsonld": "https://wpnews.pro/news/varrate-training-free-variable-rate-kv-cache-compression-for-long-context-llms.jsonld"}}