{"slug": "breaking-the-1-58-bit-barrier-for-ternary-llms", "title": "Breaking the 1.58-bit Barrier for Ternary LLMs", "summary": "A new arXiv paper (2609.16338v1) introduces BITCOS, a distribution-adaptive storage layout for ternary LLMs that costs 2 − z bits per weight given a zero density z, after measuring 29 ternary models and finding zeros account for up to 51.5% of all weights. BITCOS stores weights more compactly than the conventional five-trit packing in 26 of the 29 tested models, reaching 1.485 bits per weight on the sparsest, and delivers up to 1.28× gain over production state-of-the-art ternary matrix-vector multiplication kernels. End-to-end decode throughput improves by up to 1.18× on CPUs and 1.27× on GPUs across 5 platforms, with optimized unpacking sequences for AVX-512, AVX2, and Intel Xe2 GPUs.", "body_md": "arXiv:2609.16338v1 Announce Type: new \nAbstract: Ternary Large Language Models (LLM) store every weight as one of three symbols $\\{-1,0,+1\\}$, so the cost of a ternary model is conventionally referenced to the information-theoretic $\\log_2 3 \\approx 1.585$ bits per weight. The prevailing deployment format packs five ternary weights into one byte (five-trit packing), and due to the power-of-two group sizes used in practice this rounds up to $1.625$ bits per weight. This effective storage bit-width treats the three symbols $\\{-1,0,+1\\}$ as equiprobable. We measure the actual symbol distribution of 29 ternary LLM models and find that zeros account for up to $51.5\\%$ of all weights. Motivated by this finding, we introduce BITCOS, a simple distribution-adaptive layout comprised of a dense presence bitmap plus a compacted sign vector, and costs $2 - z$ bits per weight element given a zero density $z$ in the model's weights. BITCOS stores weights more compactly than the five-trit packing in 26 of the 29 tested models, and reaches $1.485$ bits per weight on the sparsest of them. BITCOS is amenable to efficient unpacking on modern processors and GPUs, and we present optimized unpacking sequences for AVX-512, AVX2 and Intel Xe2 GPUs. Measured against production state-of-the-art ternary matrix-vector multiplication kernels, at the zero densities real-world ternary models exhibit, the realized gain with our proposed layout is up to $1.28\\times$. Finally, we illustrate end-to-end LLM inference results on 5 different platforms (client and server CPUs, integrated and discrete Xe2 GPUs) where decode throughput improves by up to $1.18\\times$ on CPUs and $1.27\\times$ on GPUs.", "url": "https://wpnews.pro/news/breaking-the-1-58-bit-barrier-for-ternary-llms", "canonical_source": "https://arxiv.org/abs/2609.16338", "published_at": "2026-09-16 04:00:00+00:00", "updated_at": "2026-09-16 04:07:46.345022+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-research", "machine-learning"], "entities": ["BITCOS", "arXiv", "AVX-512", "AVX2", "Intel Xe2"], "alternates": {"html": "https://wpnews.pro/news/breaking-the-1-58-bit-barrier-for-ternary-llms", "markdown": "https://wpnews.pro/news/breaking-the-1-58-bit-barrier-for-ternary-llms.md", "text": "https://wpnews.pro/news/breaking-the-1-58-bit-barrier-for-ternary-llms.txt", "jsonld": "https://wpnews.pro/news/breaking-the-1-58-bit-barrier-for-ternary-llms.jsonld"}}