Compressing LLMs Without Compromise

Lossless LLM compression schemes such as DFloat11 can cut tensor data volume by an average of 1.4x while retaining all information and preserving model accuracy, offering an alternative to quantization, which degrades accuracy and requires tuning per model and use case. The schemes exploit structure in BF16 exponents, where only about 3 of the 8 bits carry useful information, and partition compressed streams into independent units so thousands of GPU threads decompress in parallel. DFloat11 applies offline Huffman coding to BF16 weight exponents for models whose weights exceed GPU memory, keeping excess tensors in CPU memory and transferring them over PCIe as needed.

LLM execution requires moving gigabytes of data, which includes static model weights, the generated KV cache, and other intermediate representations. Depending on the deployment, this may involve communication across the network, PCIe links, accelerator links e.g., NVLink https://www.nvidia.com/en-us/data-center/nvlink/ , and memory interconnects, often leaving LLM execution bottlenecked by the data movement feeding it. An obvious mitigation for this overhead is to simply reduce the amount of data transferred. Quantization is a field of research that adopts this principle, where tensors are shrunk to a smaller data format https://arxiv.org/abs/2210.17323 , reducing the space and interconnect traffic needed for inference. There’s a rich body of work examining quantization’s usefulness on different LLM components like weights https://arxiv.org/abs/2306.00978 , activations https://arxiv.org/abs/2310.19102 , attention https://arxiv.org/abs/2407.08608 , and the KV cache https://arxiv.org/abs/2303.06865 . While quantization has earned its place in the field, it is difficult to universally deploy. Model accuracy degrades https://aclanthology.org/2025.emnlp-main.479/ due to loss of information, requiring quantization to be tuned for specific models https://arxiv.org/abs/2402.18158 and use cases https://arxiv.org/abs/2504.04823 of LLMs. It has been shown that even when model accuracy appears to be unaffected, side effects manifest in unpredictable ways https://arxiv.org/abs/2407.09141 , such as increased rambling from quantized LLMs. The impact of quantization varies considerably across model architectures and workloads, making it difficult to guarantee consistency https://arxiv.org/abs/2605.15208 after deployment. Lossless compression: Reducing data without losing information Is it possible to reduce data volume without compromising on accuracy? Lossless LLM compression is an emerging line of work that demonstrates this. All information is retained, preserving model accuracy, while data volume is also reduced. While necessarily providing less compression than quantization, the reduction in volume is significant 1.4x average data reduction . Lossless compression is not without challenges. Tensors have to be reconstructed, leading decompression to lie along the critical path, potentially overriding the benefits of reduced data transfers. The floating point values contained in tensors can also be difficult to compress. For example, outlier values must be preserved, requiring additional tracking metadata. Fortunately, structure within LLM tensors provides compression opportunities. Models typically use BF16 data type, containing 1 sign bit, 8 exponent bits, and 7 mantissa bits Figure . The values of the mantissa bits follow a fairly random distribution, but the exponent bits exhibit structure. While 8 bits are provided, usually only ~3 bits carry useful information https://fergusfinn.com/blog/weight-entropy/ . The other 5 bits are low entropy. A few lossless compression schemes have emerged that take advantage of this structure, demonstrating improvements for LLM inference. These schemes are developed for GPUs, as they are typically the processor of choice for LLM execution due to their high throughput. The mechanisms described here partition the compressed stream into independent units so that thousands of GPU threads decompress in parallel, generating significant decompression throughput. The schemes minimize metadata to promote GPU cacheability and avoid extraneous overhead of metadata accesses during decompression. For example, DFloat11 https://arxiv.org/abs/2504.11651 targets cases where LLM weights exceed GPU memory capacity. Excess weight tensors are kept in CPU memory and transferred across PCIe as needed, this is known as offloading. The key idea is to use offline Huffman coding to compress the exponents ofBF16 weights, replacing these 8 bits with fewer bits that act as lookup keys. Compact lookup tables are generated that fit within the GPU’s shared memory https://developer.nvidia.com/blog/using-shared-memory-cuda-cc/ . Leveraging DFloat11 lossless compression, model sizes are reduced by ~30%, allowing a 48 GB model to fit in a 40 GB GPU. This translates to 2 – 40x faster inference over HuggingFace Transformers https://aclanthology.org/2020.emnlp-demos.6/ , as data is no longer moved across PCIe. Going further, ZipServ https://dl.acm.org/doi/abs/10.1145/3779212.3790250 investigates data movement that occurs within the GPU, specifically, when bringing model weights from the GPU memory into tensor cores before performing matrix multiplication. Similar to DFloat11, ZipServ replaces the 8 exponent bits of BF16 LLM weights with 3 bits, compressing the top 7 most frequent exponent values offline. The remaining exponents map to a special value, which indicates the exponents are stored in full form in a separate location. Fused matrix multiplication kernels fetch and decompress directly into tensor cores, avoiding the memory round trip typically required for decompression. This increases throughput by 22% on average over vLLM https://vllm.ai/ . Our work, Invariant Bit Packing https://arxiv.org/abs/2605.30728 IBP , generalizes compression and makes it data-type agnostic. It enables not only runtime decompression, but also compression, extending the use case to include the runtime-generated KV cache. IBP discovers invariant bits , i.e., repeated low-entropy bit values across tensors, and stores these bits as metadata in the GPU, small enough to fit in GPU shared memory. Identifying invariant bits across a small subset of tensors is sufficient to generalize to the full set. When compressing, these bits are removed from the tensors, while decompression adds the invariant bits back using cheap bit shift operations. IBP achieves an average of 24% faster LLM inference over FlexGen https://arxiv.org/abs/2303.06865 and InfiniGen https://arxiv.org/abs/2406.19707 , by decompressing tensors on-the-fly while transferring from the CPU. This improves both weight and KV cache transfers, demonstrating the generality of lossless compression. This topic is making its way to deployment. Cloudflare revealed Unweight https://blog.cloudflare.com/unweight-tensor-compression/ for NVIDIA H100 lossless decompression, building on the ideas previously discussed. IBM Research demonstrated ZipNN https://research.ibm.com/blog/Zip-NN-AI-compression boasting similar lossless compressibility for LLMs, targeting storage and network transfers. Doubleword.ai has shown that newer FP8 models are also compressible https://resources.doubleword.ai/resources/pushing-memory-bound-kernels-beyond-the-speed-of-light-with-lossless-decompression due to the same exponent redundancy. The next frontier: Compressing the full data path Lossless compression is having its moment for GPU data movement optimization, but the GPU is not the only place where compression can help. In scale-out LLM deployments, model and computations are split across multiple axes data, tensor, sequence, pipeline, and expert referred to as 5D parallelism https://huggingface.co/spaces/nanotron/ultrascale-playbook?section=5d parallelism in a nutshell . Each axis can add new data traffic. For example, tensor parallelism adds partial activation movement, incurring overheads of 20% https://arxiv.org/abs/2505.11329 on NVLink transfers in multi-GPU deployments. Deploying across multiple machines exacerbates this problem, as these rely on lower bandwidth network connections. Further, expert parallelism https://nvidia.github.io/TensorRT-LLM/advanced/expert-parallelism.html , where specific LLM layers are split across GPUs, requires all-to-all communication to move intermediate tokens, incurring 40% overhead https://arxiv.org/abs/2505.13345 . All of this movement could be compressed. A fundamental question is where the decompression should happen. Using the GPU for decompression can steal cycles from other concurrent compute tasks. Conversely, if we decompress early, e.g., at a network smart switch or SmartNIC, the data grows before crossing PCIe or NVLink, losing benefits for downstream links. Depending on the deployment, either approach may have acceptable tradeoffs. NVIDIA B200 GPUs demonstrate decompression along the transfer path by including a new hardware decompression engine https://developer.nvidia.com/blog/speeding-up-data-decompression-with-nvcomp-and-the-nvidia-blackwell-decompression-engine/ that transfers compressed data across PCIe and then decompresses without utilizing GPU multiprocessors. It currently supports a limited range of algorithms LZ4/Snappy/Deflate . How to extend this range and make the engine flexible, is an open question. Hardware architects have other potential substrates for decompression present within the memory system: processing in or near memory can store data in compressed form in memory, then decompress data as it leaves memory, increasing effective GPU memory capacity without reserving GPU cores for decompression. This allows more data to be kept within GPU memory, conversely reducing the volume of data needed to be transferred across low bandwidth links like PCIe. Adapting lossless compression for dynamically changing data is another significant open problem. All the schemes we discussed target static data. This works well for inference, where data values e.g., KV cache entries or weight tensors are reused across iterations, amortizing compression overhead. Training breaks this assumption. The entire working set changes every iteration, needing repeated loops of compression and decompression. Mitigating this overhead requires lossless compression that can quickly recompress data. More ambitiously, a scheme that supports tensor updates e.g., gradient descent https://en.wikipedia.org/wiki/Gradient descent on compressed data could eschew the need for recompression entirely. Conclusion Lossless compression for LLMs is still in its infancy, but already showing promise. Existing work has demonstrated that LLM tensors can be compressed by ~30%. The overhead of communication in multi-GPU deployments is significant 20 – 40% , providing a prime future use case. However, it requires tailoring compression to the intricacies of the intervening links, and involves identifying the best candidate to perform decompression. This is an open research field ripe for investigation. About the Authors: Aditya K Kamath http://akkamath.github.io was a Ph.D. student in Computer Science and Engineering at the University of Washington UW . His work revolved around analyzing data movement and mitigating its performance impact in memory-intensive applications. Simon Peter https://homes.cs.washington.edu/~simpeter/ is an associate professor in Computer Science and Engineering at UW. His work is in low latency, scalable, and energy-efficient data center systems and cloud application design, across the hardware/software continuum. Disclaimer: These posts are written by individual contributors to share their thoughts on the Computer Architecture Today blog for the benefit of the community. Any views or opinions represented in this blog are personal, belong solely to the blog author and do not represent those of ACM SIGARCH or its parent organization, ACM.