Photo: Logan Wallace / The Ohio State University/Logan
A new technique from University of Edinburgh and NVIDIA compresses AI memory by 8x while actually boosting performance on math, science, and coding tests
Making something smaller and better at the same time sounds like a violation of basic physics. But researchers at the University of Edinburgh, working with NVIDIA, have pulled it off with large language models. Their technique, called Dynamic Memory Sparsification (DMS), compresses a critical piece of AI infrastructure by 8x while somehow making the models score higher on difficult benchmarks.
If AI models can run just as well, or better, with a fraction of the memory, the door opens to deploying serious reasoning capabilities on devices that currently can’t handle them, including wearables, smart home hardware, and edge devices.
How DMS actually works #
To understand the breakthrough, you need to know about the key-value (KV) cache. When an AI model reasons through a problem, it stores intermediate results in this cache, essentially a working memory that lets the model keep track of its own thought process. The longer and more complex the reasoning chain, the bigger that cache grows, and the more computational resources it demands.
DMS takes a scalpel to this process. Instead of keeping every token in the cache, it selectively retains only the ones that matter most and drops the rest. The result is a KV cache compressed to one-eighth its original size. By clearing out the noise, the models can explore deeper and more intricate reasoning pathways within the same computational budget.
The benchmark results #
On AIME 24, a math olympiad qualifying exam, compressed models using DMS scored an average of 12 points higher than their uncompressed counterparts.
On GPQA Diamond, a benchmark built from graduate-level science problems across physics, chemistry, and biology, the DMS-compressed models posted scores more than 8 points higher on average.
On LiveCode Bench, which tests practical programming ability, compressed models gained 10 points over their full-cache equivalents while reading the same amount of KV cache data.
Lead researcher Dr. Edoardo Ponti summarized the dual benefit simply.
“Models can reason faster but with the same quality.”
Within a fixed time window, an LLM using DMS can explore more reasoning threads and arrive at stronger conclusions.
A longer trend, accelerating #
The AI field has been chasing model efficiency since the mid-2010s, when techniques like knowledge distillation, pruning, and quantization started proving that smaller models could compete with larger ones on specific tasks. What DMS adds to this lineage is a focus on inference-time memory, the resources consumed not when a model is being trained, but when it’s actually being used. Compressing inference memory by 8x means every deployment becomes dramatically cheaper to run.
The research was presented at the NeurIPS conference and detailed in a paper titled “Inference-Time Hyper-Scaling with KV Cache Compression.” Evaluations were conducted using Llama and Qwen models.
Disclosure: This article was edited by Editorial Team. For more information on how we create and review content, see our