Transformer Efficiency: A Closer Look at KV Cache Compression
New research characterizes the intrinsic compressibility of KV caches in Transformer models, proposing a principled algorithm for efficient inference. The study uses minimax risk to determine when accurate compression is…