Grouped Value Attention: Efficient KV Caching via On-Demand Key Reconstruction

Researchers introduced Grouped Value Attention, a method that reduces the Transformer KV cache by reconstructing keys on demand rather than storing them at every decoding step. The technique builds on grouped-query attention (GQA), which already cuts cost by sharing key-value heads but still stores both a key and a value at each step; Grouped Value Attention instead stores only values and reconstructs keys as needed. The KV cache is a primary bottleneck for Transformer decoding because its memory footprint and cache-read traffic grow with sequence length.

The KV cache is a primary bottleneck for Transformer decoding: its memory footprint and cache-read traffic grow with sequence length. Grouped-query attention GQA reduces this cost by sharing key-value heads, but still stores both a key and a value at every step. We introduce Grouped Value Attentio