{"slug": "semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff", "title": "SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference", "summary": "A new arXiv preprint (2608.28911v1) introduces SemKV, a semantic mixed-precision KV cache quantization method that achieves a measured 6.0x storage reduction for long-context LLM inference with no statistically detectable quality loss, and up to 7.9x when combined with a distortion-optimized quantizer. The researchers identified a 'quality cliff' between 2.0 and 2.322 code bits per value for uniform quantization, below which performance collapses, and SemKV avoids this by assigning two adjacent above-cliff precisions based on model-internal token importance scores.", "body_md": "arXiv:2608.28911v1 Announce Type: new\nAbstract: The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.", "url": "https://wpnews.pro/news/semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff", "canonical_source": "https://arxiv.org/abs/2608.28911", "published_at": "2026-09-01 04:00:00+00:00", "updated_at": "2026-09-01 04:25:36.918315+00:00", "lang": "en", "topics": ["large-language-models", "ai-research", "ai-infrastructure"], "entities": ["arXiv", "SemKV", "Llama-3.1-8B-Instruct", "Mistral-7B", "TurboQuant-MSE"], "alternates": {"html": "https://wpnews.pro/news/semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff", "markdown": "https://wpnews.pro/news/semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff.md", "text": "https://wpnews.pro/news/semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff.txt", "jsonld": "https://wpnews.pro/news/semkv-semantic-mixed-precision-kv-cache-quantization-guided-by-the-quality-cliff.jsonld"}}