{"slug": "kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei", "title": "KVarN: Native vLLM KV-cache quantization back end by Huawei", "summary": "Huawei released KVarN, a native KV-cache quantization back end for vLLM that delivers up to 5x more cache capacity and 1.3x the throughput of FP16 while maintaining FP16-level accuracy. The calibration-free system handles agentic and long-context workloads by quantizing keys at 4 bits and values at 2 bits, outperforming existing methods like TurboQuant by up to 2.4x in throughput with higher accuracy. KVarN ships as a vLLM fork that requires only a single flag change to enable, eliminating the need for model modifications or calibration data.", "body_md": "⚡️\n\nBuilt for agentic and long-context workloads.\n\n💡 KVarN delivers\n\n3-5x more KV-cache capacityandup to ~1.3x the throughputof FP16, so you fit far longer contexts and serve more concurrent requests, withFP16-level accuracy.\n\n🔌\n\nCalibration-free, plug-and-play with vLLM.A native vLLM attention backend: add one flag, no model changes, no calibration.\n\n🥊\n\nUp to ~2.4× TurboQuant throughput, same capacity,higher accuracy.\n\nkvarn/kvɑːɳ/ ·noun(Swedish)\n\n- A grinding apparatus used to reduce substances into smaller particles or powder, especially grains, seeds, spices, coffee beans, KV-caches.\n\nKV-cache quantization usually comes with a catch. As the\n[vLLM TurboQuant blog](https://vllm.ai/blog/2026-05-11-turboquant) shows, existing\nmethods buy extra KV-cache capacity but **give up throughput** (TurboQuant reports\n**40 to 52% lower throughput** for 2.3-3.7x capacity), and aggressive low-bit\nquantization also tends to **cost accuracy**. Losing both speed *and* quality is\nthe main reason KV-cache quantization is rarely turned on in production.\n\n**KVarN is built to keep both.** On Qwen3-32B (AIME25, 16K-context burst, TP=2) it\nmatches FP16 accuracy and **beats its throughput** while delivering ~4× the KV-cache capacity:\n\nKVarN stays in the upper-right corner the blog's methods can't reach: **FP16-level\naccuracy, FP16-or-better throughput, and several times the context.**\n\nKVarN ships as a vLLM fork. Install it like vLLM, then select the KVarN KV-cache dtype.\n\n```\n# 1. Clone\ngit clone https://github.com/huawei-csl/KVarN.git\ncd KVarN\n\n# 2. Install (uses the upstream precompiled wheel; KVarN kernels are Triton, JIT-compiled at runtime)\nVLLM_USE_PRECOMPILED=1 pip install -e .\npython\nfrom vllm import LLM, SamplingParams\n\nllm = LLM(\n    model=\"Qwen/Qwen3-32B\",\n    dtype=\"float16\",                    # KVarN runs in float16\n    kv_cache_dtype=\"kvarn_k4v2_g128\",   # enable KVarN\n    block_size=128,                     # KVarN tile size\n)\nprint(llm.generate(\"Explain KV-cache quantization in one sentence.\",\n                    SamplingParams(max_tokens=64))[0].outputs[0].text)\n```\n\nServing works the same way:\n\n```\nvllm serve Qwen/Qwen3-32B --dtype float16 --kv-cache-dtype kvarn_k4v2_g128 --block-size 128\n```\n\nNote:KVarN runs in`float16`\n\ncompute. The tile / page size is currently fixed at 128 (one vLLM block = one KVarN tile); other page sizes are coming soon.\n\nTip (capacity):KVarN realizes its full KV-cache capacity when there is room to amortize a small fixed decode workspace. On multi-GPU or generous`--gpu-memory-utilization`\n\nsetups this is automatic. On a tight single-GPU budget, vLLM's CUDA-graph memory profiler can over-reserve and shrink the KV pool; set`VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0`\n\n(and/or raise`--gpu-memory-utilization`\n\n) to recover the full capacity.\n\nKVarN quantizes the KV cache one fixed-size token tile at a time, walking each tile through the four stages above:\n\n-\n**Cache**: the raw fp16 KV tile (channels × tokens), straight from attention. -\n**Rotated Cache**: a** Hadamard rotation**along the channel dimension mixes channels so that per-channel outliers are spread out, making the tile easier to quantize. The rotation is orthonormal, so attention scores are preserved. -\n**Normalized Cache**:** iterative variance normalization**(Sinkhorn-like) alternates column- and row-wise standard-deviation normalization in log space, equalizing variance across the tile and shrinking quantization error before any rounding happens. -\n**Quantized Cache**:** asymmetric round-to-nearest**at low bit-width, with the scales folded back in at read time (keys per channel, values per token).\n\nThe shipped preset spends **more bits on keys than values** (`kvarn_k4v2_g128`\n\n:\n4-bit keys, 2-bit values). We chose to release this configuration because it meets\nthe strictest accuracy bar, matching FP16, that the most demanding production\ndeployments and vLLM require, while still delivering throughput above FP16.\n\nKVarN is the official vLLM implementation of our paper:\n\n📄\n\nKVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks([arXiv:2606.03458])\n\nIf you use KVarN, please cite:\n\n```\n@misc{muller2026kvarn,\n      title={KVarN: Variance-Normalized KV-Cache Quantization Mitigates Error Accumulation in Reasoning Tasks}, \n      author={Lorenz K. Muller and Philippe Bich and Chiara Boretti and Hyun-Min Chang and Jiawei Zhuang and Lukas Cavigelli},\n      year={2026},\n      eprint={2606.03458},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG},\n      url={http://arxiv.org/abs/2606.03458}\n}\n```\n\nKVarN is built on [vLLM](https://github.com/vllm-project/vllm) (v0.22.0) and is\nreleased under the Apache 2.0 License. The original vLLM README is preserved as\n[ README_vLLM.md](/huawei-csl/KVarN/blob/main/README_vLLM.md).", "url": "https://wpnews.pro/news/kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei", "canonical_source": "https://github.com/huawei-csl/KVarN", "published_at": "2026-06-04 15:18:00+00:00", "updated_at": "2026-06-04 15:47:54.494214+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "ai-research", "ai-products"], "entities": ["Huawei", "vLLM", "KVarN", "TurboQuant", "Qwen3-32B", "AIME25"], "alternates": {"html": "https://wpnews.pro/news/kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei", "markdown": "https://wpnews.pro/news/kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei.md", "text": "https://wpnews.pro/news/kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei.txt", "jsonld": "https://wpnews.pro/news/kvarn-native-vllm-kv-cache-quantization-back-end-by-huawei.jsonld"}}