{"slug": "selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention", "title": "SelKV: Selective KV Cache Merging with Per-Token Merge-or-Drop and Attention Compensation", "summary": "Researchers propose SelKV, a training-free framework for compressing the key-value (KV) cache in large language models (LLMs) that uses a soft cosine gate to selectively merge or discard tokens and an attention-ratio compensation mechanism to correct softmax imbalance. Evaluated on LongBench while retaining only 25% of the KV cache, SelKV achieves strong compressed performance against baselines, maintains near-lossless quality on grouped-query attention (GQA) models, outperforms the full-cache baseline on complex multi-document QA tasks, and delivers a 3.3x decoding speedup at 100k tokens.", "body_md": "arXiv:2607.16213v1 Announce Type: new\nAbstract: Large Language Models (LLMs) generate text autoregressively, relying on a key-value (KV) cache whose memory footprint grows linearly with context length, creating a major bottleneck. Recent compression methods mitigate this cost via token merging; however, these approaches often rely on indiscriminate aggregation, which degrades representations and introduces attention sag, a mismatch where merged tokens receive the same softmax mass as individual tokens despite encoding multiple inputs. We propose a training-free, dual-component framework for KV cache compression that addresses these limitations. First, a soft cosine gate adaptively modulates merging decisions based on value-vector similarity, suppressing or discarding dissimilar tokens to preserve semantic fidelity. Second, we introduce an attention-ratio compensation mechanism that applies a decoding-time logit bias derived from prefill attention statistics, correcting the softmax imbalance induced by merging. Evaluated on LongBench (16 English datasets) while retaining only 25% of the KV cache, our framework achieves strong compressed performance against representative one-shot baselines. It is especially robust on the evaluated grouped-query attention (GQA) models, maintaining nearlossless generation quality. Furthermore, the method outperforms the full-cache baseline on complex multi-document QA tasks and delivers a 3.3x decoding speedup at 100k tokens.", "url": "https://wpnews.pro/news/selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention", "canonical_source": "https://arxiv.org/abs/2607.16213", "published_at": "2026-07-21 04:00:00+00:00", "updated_at": "2026-07-21 04:07:38.605487+00:00", "lang": "en", "topics": ["large-language-models", "artificial-intelligence", "machine-learning"], "entities": ["SelKV", "LongBench", "arXiv"], "alternates": {"html": "https://wpnews.pro/news/selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention", "markdown": "https://wpnews.pro/news/selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention.md", "text": "https://wpnews.pro/news/selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention.txt", "jsonld": "https://wpnews.pro/news/selkv-selective-kv-cache-merging-with-per-token-merge-or-drop-and-attention.jsonld"}}