{"slug": "a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a", "title": "A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint", "summary": "A new arXiv paper (2608.28003v1) proposes a layer bit-width allocation method for Gemma-3-1B that maximizes latency reduction under a quality-degradation constraint, achieving an 11.0% latency reduction with negligible quality loss (98.90% Top-1 agreement, +0.85% perplexity degradation) on an RTX 5090. The method, built on the prior SA-PTQ sensitivity profile and implemented in TensorRT-LLM, outperforms uniform quantization approaches like GPTQ and AWQ, and can reach up to 19.1% speedup with acceptable quality loss.", "body_md": "arXiv:2608.28003v1 Announce Type: new\nAbstract: This paper proposes a layer bit allocation method for Gemma-3-1B, formulating the problem as performance maximization (latency decrease) given a degradation budget constraint (allowable level of generation quality loss). This approach is different from time- and resource-consuming uniform layer quantization methods that are used in the literature (like GPTQ or AWQ) or allocation methods without proven performance-accelerating effect (like MixLLM or TorchAO). The layer sensitivity profile resulting from our prior work SA-PTQ is applied using the activation pass-through mode inside TensorRT-LLM. For each layer precision is determined individually in blocks, according to a grouping introduced in the prior step (5+5, 10+10, all26), differentiating the contribution of FFN, Attention, and lm_head to the overall speedup. The clock speed was measured for 13 W8A8 variants on an RTX 5090. We find that for FFN and lm_head the time cost of quantization/dequantization is compensated for by the use of integer arithmetic, while for short context lengths, the opposite holds true for Attention: an additional step of quantization slows execution down. We propose a manual implementation of SmoothQuant for TensorRT-LLM which was necessary due to export failures, unavailable for lm_head. The best solution found under joint consideration of all three criteria with minimal degradation was FFN 5+5 with lm_head, providing an 11.0% reduction in latency with negligible quality loss (98.90% Top-1 agreement, +0.85% perplexity degradation). With acceptable quality loss for FFN all26 + lm_head, a speedup up to 19.1% was found possible. We suggest further optimizations: fused attention kernels in INT8, KV-cache quantization, using FP8 instead of INT8 and partial Attention quantization analogous to FFN.", "url": "https://wpnews.pro/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a", "canonical_source": "https://www.machinebrief.com/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-nlp0", "published_at": "2026-08-31 04:00:00+00:00", "updated_at": "2026-08-31 04:51:58.100941+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-research", "ai-infrastructure"], "entities": ["Gemma-3-1B", "TensorRT-LLM", "RTX 5090", "GPTQ", "AWQ", "MixLLM", "TorchAO", "SmoothQuant"], "alternates": {"html": "https://wpnews.pro/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a", "markdown": "https://wpnews.pro/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a.md", "text": "https://wpnews.pro/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a.txt", "jsonld": "https://wpnews.pro/news/a-method-for-layer-bit-width-allocation-in-llm-quantization-via-performance-a.jsonld"}}