SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation

Researchers propose SelKV, a training-free framework for compressing the key-value (KV) cache in large language models (LLMs) that uses a soft cosine gate to selectively merge or discard tokens and an attention-ratio compensation mechanism to correct softmax imbalance. Evaluated on LongBench while retaining only 25% of the KV cache, SelKV achieves strong compressed performance against baselines, maintains near-lossless quality on grouped-query attention (GQA) models, outperforms the full-cache baseline on complex multi-document QA tasks, and delivers a 3.3x decoding speedup at 100k tokens.

arXiv:2607.16213v1 Announce Type: new Abstract: Large Language Models LLMs generate text autoregressively, relying on a key-value KV cache whose memory footprint grows linearly with context length, creating a major bottleneck. Recent compression methods mitigate this cost via token merging; however, these approaches often rely on indiscriminate aggregation, which degrades representations and introduces attention sag, a mismatch where merged tokens receive the same softmax mass as individual tokens despite encoding multiple inputs. We propose a training-free, dual-component framework for KV cache compression that addresses these limitations. First, a soft cosine gate adaptively modulates merging decisions based on value-vector similarity, suppressing or discarding dissimilar tokens to preserve semantic fidelity. Second, we introduce an attention-ratio compensation mechanism that applies a decoding-time logit bias derived from prefill attention statistics, correcting the softmax imbalance induced by merging. Evaluated on LongBench 16 English datasets while retaining only 25% of the KV cache, our framework achieves strong compressed performance against representative one-shot baselines. It is especially robust on the evaluated grouped-query attention GQA models, maintaining nearlossless generation quality. Furthermore, the method outperforms the full-cache baseline on complex multi-document QA tasks and delivers a 3.3x decoding speedup at 100k tokens.