Researchers have introduced Faster Flash Decoding (FFD), a hardware-algorithm co-design framework that achieves up to 11.6x kernel-level speedup and 2.37x end-to-end throughput improvement for long-co…
Researchers propose PuzzleKV, a training- and calibration-free method for KV cache compression in large language models, which partitions per-head KV caches into fixed-length logical pages and applies…
Researchers introduced Attention-Aware Transform Coding (AATC), a KV cache compression method that reduces memory use by approximately 5.8x while maintaining near-lossless accuracy on models like Llam…
Researchers introduced Memory-Anchor Routing across Context History (MARCH), a recurrent architecture that scales state-space models beyond fixed-size memory by periodically caching state anchors with…
Researchers propose SelKV, a training-free framework for compressing the key-value (KV) cache in large language models (LLMs) that uses a soft cosine gate to selectively merge or discard tokens and an…
VarRate, a training-free variable-rate KV cache compression method for long-context LLMs, outperforms token-selection methods by allocating low-rank budgets based on query salience instead of evicting…
New research characterizes the intrinsic compressibility of KV caches in Transformer models, proposing a principled algorithm for efficient inference. The study uses minimax risk to determine when acc…
A new benchmark evaluates KV-cache optimization techniques—quantization, pruning, and merging—for long-context LLM serving, finding that compression ratio alone poorly predicts end-to-end performance.…
Researchers introduced Simplified Sparse Attention (SSA), a method that uses gist tokens to achieve sparse attention without architectural changes, outperforming baselines on LongBench and improving r…
A 1 million token context window allows an LLM to receive vast input but does not guarantee accurate memory, retrieval, or reasoning, as models often fail on mid-document facts and multi-hop connectio…
Researchers introduced Hankel Reduced order Model (HRM) adapters, a state space model-based fine-tuning method that outperforms LoRA on long-context tasks like QuALITY and QMSum by leveraging FFT-base…
Researchers propose Dustin, a sparse verification framework for long-context speculative decoding in LLMs, achieving a 27.85x speedup in self-attention and 9.17x end-to-end decoding speedup at 32k seq…
Researchers introduced Still, a per-layer Perceiver model that compacts KV cache in a single forward pass, enabling efficient long-context language model deployment. On Qwen and Gemma models, Still ou…