WakeKV, a reactive KV-cache residency policy, improves miss rate over frozen head classification and destructive eviction at matched memory or budget, according to an arXiv paper (2610.02713v1). Acros…
Anthropic's Jacobian lens, introduced in a July 6 paper, reveals a low-dimensional 'J-space' inside language models that encodes tokens the model can verbalize. Developer Anyesh repurposed this signal…
VarRate, a training-free variable-rate KV cache compression method for long-context LLMs, outperforms token-selection methods by allocating low-rank budgets based on query salience instead of evicting…
A new benchmark evaluates KV-cache optimization techniques—quantization, pruning, and merging—for long-context LLM serving, finding that compression ratio alone poorly predicts end-to-end performance.…