Compression Does Not End at Selection: Post-Selection Positional Geometry for Sparse KV Caches

RoPERelay, a post-selection positional reorganization method for sparse KV caches, improved answer quality in 32 of 36 comparisons across two models, six sparse selectors, and three long-term memory tasks, according to an arXiv paper (arXiv:2610.21126v1). On LongMemEval-S with Llama-3.1-8B-Instruct and BM25, accuracy rose from 23.6% to 35.4%, with measured complete-request time overhead of approximately 1%-3% in the evaluated configurations. The method assigns compact order-preserving RoPE coordinates, relocates cached key phases through inverse rotation and re-rotation, and synchronizes the current query and subsequent generation without renewed selection, historical re-prefill, or additional cache budget.

arXiv:2610.21126v1 Announce Type: new Abstract: Sparse KV compression reduces long-context inference costs by retaining selected historical states, yet these states typically keep their coordinates from the full history. Under Rotary Position Embedding RoPE , gaps left by evicted tokens continue to affect query-key matching, potentially limiting the use of retained evidence. We study post-selection positional geometry on a fixed selector-produced cache, examining how positional organization affects the use of retained information and which coordinate and continuation rules improve answer quality. To reorganize this geometry, we propose RoPERelay, which assigns compact, order-preserving RoPE coordinates, relocates cached key phases through inverse rotation and re-rotation, and synchronizes the current query and subsequent generation. It reuses existing K/V without renewed selection, historical re-prefill, or additional cache budget. Across 36 comparisons spanning two models, six sparse selectors, and three long-term memory tasks, RoPERelay improves answer quality in 32 cases. For Llama-3.1-8B-Instruct with BM25 on LongMemEval-S, accuracy increases from 23.6% to 35.4%. Measured complete-request time overhead is approximately 1%-3% in the evaluated configurations. These findings establish positional organization as a distinct optimization dimension for sparse KV caches after content selection.