Grouped Value Attention slashes transformer KV memory by roughly 45 % without hurting benchmark scores. By storing only grouped values and reconstructing keys on the fly, it eliminates the need to materialize a full key matrix during decode, cutting the persistent cache scalars dramatically.
Before these works, long‑context decoding relied on full key–value caches that grew linearly with sequence length. Even techniques like grouped‑query attention still kept a key for every token, so memory pressure remained the dominant bottleneck in autoregressive inference.
On a 350 M‑parameter model, GVA cuts persistent cache scalars by about 46 % while keeping average task accuracy within 0.01 points of GQA. The authors report “this representation reduces persistent cache scalars by approximately 45–47 % relative to matched GQA” and note that “the 16‑dimensional positional variant reaches 44.35 average accuracy across five tasks, compared with 44.36 for GQA and 43.88 for MLA” [1].
DeepSeek‑V4.1‑Flash pushes KV compression further, shrinking the global HBM footprint to 890 bytes per token – roughly one quarter of its predecessor. Moreover, “through a dedicated deployment optimization known as SWA Bounded Replay … reduces its persistent KV cache footprint … to roughly 1/8 of that of DeepSeek‑V4‑Flash,” and despite these cuts the model “delivers substantially better performance than the baseline” [2].
Both papers leave open how these memory savings translate into end‑to‑end latency reductions on commodity GPUs. GVA’s custom decoding kernels are still under evaluation, so real‑world throughput gains remain unproven; DeepSeek’s design includes FP4 caching and cross‑layer reuse, which could benefit from hardware supporting low‑precision arithmetic.
If the reported cuts hold in production, the KV cache allocation for decoder‑only models could potentially be reduced substantially while maintaining quality. These reductions suggest that re‑evaluating long‑context benchmarks with smaller cache budgets may enable longer token contexts on more modest hardware.