Beyond Scalar Sensitivity: Activation-Aware Mixed-Precision LLM Quantization with Cross-Layer Refinement

Researchers proposed Cross-layer Activation-aware Sensitivity Allocation (CASA), a two-phase mixed-precision quantization method that replaces scalar sensitivity proxies with an activation-aware metric derived from the Kronecker-factored Hessian, according to a new arXiv paper (2609.25916v1). The authors prove that even the optimal scalar proxy incurs multiplicative distortion up to the square root of the product of the input- and output-side Hessian condition numbers, a bound ranging from 10^1 to 10^13 for typical LLM modules, which makes inter-module sensitivity ranking unreliable. Experiments across multiple LLMs and bit budgets show CASA achieves lower perplexity than the latest scalar-proxy baselines, especially below 3 bits per weight, with zero-shot accuracy gains tracking each model's average per-module condition number.

arXiv:2609.25916v1 Announce Type: new Abstract: Mixed-precision weight quantization is commonly formulated as a Multiple-Choice Knapsack Problem MCKP , yet existing solvers rely on scalar sensitivity proxies that collapse each weight matrix's Hessian into a single number and treat every module independently. We prove that even the optimal scalar proxy incurs multiplicative distortion up to $\sqrt{\kappa \mathbf{A} \kappa \mathbf{B} }$ relative to the full activation-aware quadratic, where $\kappa \mathbf{A} $ and $\kappa \mathbf{B} $ denote the condition numbers of the input- and output-side Hessian factors. This bound varies from $10^1$ to $10^{13}$ for typical LLM modules, making inter-module sensitivity ranking unreliable. To address these limitations, we propose Cross-layer Activation-aware Sensitivity Allocation CASA , a two-phase method. In Stage 1, the scalar proxy is replaced by an activation-aware metric derived from the Kronecker-factored Hessian, reducing the MCKP to a form whose continuous relaxation admits a closed-form solution. In Stage 2, a cross-layer-aware local search evaluates bit-width updates using the end-to-end model loss. Experiments on multiple LLMs across different bit budgets show that CASA achieves lower perplexity than the latest scalar-proxy baselines, especially at ultra-low bit-widths $<3$ bits per weight . Moreover, the performance gain in zero-shot accuracy tracks the per-model average condition-number over modules, confirming the distortion bound as a practical indicator of scalar-proxy failure.