VarRate: Training-Free Variable-Rate KV Cache Compression for Long-Context LLMs

VarRate, a training-free variable-rate KV cache compression method for long-context LLMs, outperforms token-selection methods by allocating low-rank budgets based on query salience instead of evicting tokens, degrading by only 3.5-5.5 points where query-aware selection collapses by 11-15 points. At a matched 20% budget on LongBench, VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B, and matches KVzip in accuracy with one-eighth the prefill overhead.

arXiv:2607.15498v1 Announce Type: new Abstract: The key-value KV cache is the main memory bottleneck in long-context large language model LLM inference. Two leading training-free families are both structurally limited: token-selection methods SnapKV, Ada-KV score importance from an observation window and evict low-scoring tokens, but eviction is irreversible -- so when the importance signal degrades under query-agnostic reuse, accuracy collapses by 11-15 points; uniform low-rank coding keeps every token but spends equal rank everywhere, wasting budget. We observe that both failures share one cure: rank should be allocated, not evicted. We present VarRate, a training-free KV codec that assigns each token a variable low-rank budget by its query salience, keeping every token at a nonzero rank. Comparable adaptive-rank codecs reach this allocation only through training; VarRate requires none. Because no token is dropped, it degrades by only 3.5-5.5 points where query-aware selection collapses. At a matched 20% budget on LongBench 16 tasks , VarRate stays within 0.8 points of the uncompressed model on both Llama-3.1-8B and Qwen2.5-7B. Averaged over the two, it is the strongest matched-memory compressor. It significantly beats its uniform-rank ablation on both models. Against KVzip, a method purpose-built for query-agnostic reuse, it is accuracy-equivalent in three of four settings and within a point overall, at about one-eighth the prefill overhead.