Researchers propose SelKV, a training-free framework for compressing the key-value (KV) cache in large language models (LLMs) that uses a soft cosine gate to selectively merge or discard tokens and an…
VarRate, a training-free variable-rate KV cache compression method for long-context LLMs, outperforms token-selection methods by allocating low-rank budgets based on query salience instead of evicting…
New research characterizes the intrinsic compressibility of KV caches in Transformer models, proposing a principled algorithm for efficient inference. The study uses minimax risk to determine when acc…
A new benchmark evaluates KV-cache optimization techniques—quantization, pruning, and merging—for long-context LLM serving, finding that compression ratio alone poorly predicts end-to-end performance.…
Researchers introduced Simplified Sparse Attention (SSA), a method that uses gist tokens to achieve sparse attention without architectural changes, outperforming baselines on LongBench and improving r…
A 1 million token context window allows an LLM to receive vast input but does not guarantee accurate memory, retrieval, or reasoning, as models often fail on mid-document facts and multi-hop connectio…
Researchers introduced Hankel Reduced order Model (HRM) adapters, a state space model-based fine-tuning method that outperforms LoRA on long-context tasks like QuALITY and QMSum by leveraging FFT-base…
Researchers propose Dustin, a sparse verification framework for long-context speculative decoding in LLMs, achieving a 27.85x speedup in self-attention and 9.17x end-to-end decoding speedup at 32k seq…
Researchers introduced Still, a per-layer Perceiver model that compacts KV cache in a single forward pass, enabling efficient long-context language model deployment. On Qwen and Gemma models, Still ou…