Visual contrast between the destructive physical slicing of high-precision post-training models versus the pristine stability of native 1-bit gold designs.
Imagine hiring a world-class tailor to sew a bespoke, double-breasted suit out of the finest Italian cashmere, only to watch them run it through an industrial woodchipper and attempt to stitch the remaining fibers back together using plastic zip-ties. It sounds absurd, yet this is precisely the computational violence we visit upon our largest generative models every single day. We spend millions of dollars training massive large language models in pristine, high-precision floating-point formats, only to aggressively slice, dice, and squeeze them into low-bit integer grids right before deployment. We call it “post-training compression,” but in reality, we are trying to fit the vast complexity of human language into a digital straightjacket, and the structural seams of our models are violently tearing under the pressure.
📊 Executive Summary:Post-Training Quantization (PTQ) breaks under extreme low-bit constraints due to dynamic activation outliers scaling beyond 1⁰⁵. This essay demonstrates that native hardware-software co-design via 1.58-bit ternary networks (BitNet b1.58) and 1.25-bit architectures (Sherry) achieves lossless inference (Huang et al., 2026; Ma et al., 2024). Integrating Walsh-Hadamard rotational transformations (Y = X × (H ÷ √N)) and asymmetric, norm-balanced KV cache compression (OScaR/KIVI) completely bypasses representation collapse, delivering up to 3.0x decoding speedups (Ashkboos et al., 2024; Liu et al., 2024; Su et al., 2026).
This late-stage slicing persists because we are desperately trying to escape the tyrannical reality of the “Memory Wall” (Wang et al., 2024; Wang et al., 2025). In autoregressive decoding, every time a model generates a single token, it must stream its entire parameter matrix from High-Bandwidth Memory (HBM) into the processor’s active registers (Liu et al., 2024; Wang et al., 2025). We spend vastly more physical energy and clock cycles moving weights across silicon than we do on the actual arithmetic calculations (Wang et al., 2024; Wang et al., 2025). It is a massive hardware bottleneck, and memory bandwidth has become the ultimate gatekeeper of inference latency (Wang et al., 2025). To break this bottleneck, developers look at floating-point weights and assume simple rounding can save the day. However, this naive approach crashes headfirst into a microscopic disrupter: the spontaneous emergence of activation outliers (Dettmers et al., 2022).
During the pre-training phase of massive transformers, a highly systematic, latent phenomenon occurs within the model’s deep layers (Dettmers et al., 2022). A microscopic fraction of dimensions within the high-dimensional latent representation space suddenly scales past magnitudes of 1⁰⁵ (Dettmers et al., 2022; Su et al., 2026). These outliers are not mathematical noise; they are structural anchor points that encode crucial syntactic rules and zero-shot reasoning capabilities (Dettmers et al., 2022). If we clip them to make quantization easier, the model’s intelligence immediately collapses (Dettmers et al., 2022; Xiao et al., 2023). If we keep them, they distort the entire quantization scale, permanently destroying the resolution of the remaining 99.9% of normal features (Dettmers et al., 2022). Post-training quantization is a temporary, structurally flawed patch; true, cost-effective scaling requires moving entirely toward native, hardware-software co-designed low-bit architectures (Ma et al., 2024).
To understand why post-training quantization fails so spectacularly at low bit-widths, we must look at the mathematical mechanics of the “squashing” effect. Traditional absolute maximum (absmax) scaling maps continuous floating-point values to discrete low-bit integer grids using a relatively simple transformation:
q = round(x × (q_max ÷ max(|x|)))
In this formulation, x represents the incoming floating-point activation, while q_max is the maximum integer boundary of our target grid (Xiao et al., 2023). For example, in a signed 4-bit representation, q_max is strictly bounded at 7. The scale factor is dictated entirely by max(|x|), the absolute maximum value present in the entire tensor (Dettmers et al., 2022). If a single activation outlier of magnitude 1⁰⁵ emerges in a layer where the normal, non-outlier weights sit quietly at 0.1, the resulting scale factor becomes microscopic (Dettmers et al., 2022). When normal features are multiplied by this tiny scale factor and rounded, they are instantly squashed to absolute zero in the quantized space (Dettmers et al., 2022; Xiao et al., 2023). The delicate semantic variations that allow an LLM to distinguish subtle nuances in text are permanently erased (Ma et al., 2024).
[Continuous Latent Space] ──► [Absmax Scaling: max(|x|) = 10⁵] ──► [Quantized Integer Grid] - Outlier dimension: 10⁵ - Outlier: mapped to 7 - Normal dimensions: 0.1 - Normal: mapped to 0 (Erased!)
This dynamic exposes a severe asymmetric shift budget between static weights and dynamic activations (Xiao et al., 2023). Static weight outliers are fixed properties of the pre-trained checkpoint and can be analyzed, pre-clipped, or smoothed offline before the model ever sees a user query (Xiao et al., 2023). Activations, however, are dynamic beasts generated at runtime in response to specific, unpredictable input sequences (Dettmers et al., 2022; Xiao et al., 2023). Because these dynamic outliers cannot be predicted ahead of time, any attempt to apply rigid, static quantization parameters across runtimes results in catastrophic representational collapse (Dettmers et al., 2022; Xiao et al., 2023). The geometric diversity of the network’s internal representations is completely neutralized, causing the model’s output perplexity to degrade to random chance levels (Huang et al., 2026).
Furthermore, quantization errors in deep neural networks do not occur in isolated silos (Huang et al., 2026). The mathematical error introduced in the first self-attention block propagates, amplifies, and multiplies exponentially as it travels through dozens of successive transformer layers (Huang et al., 2026). Empirical studies show that uncorrected 1-bit models suffer from catastrophic perplexity explosions exceeding 1⁰⁸ on standard benchmarks like WikiText-2 (Huang et al., 2026). The model essentially loses its ability to form coherent syntax, babbling incoherent characters as the cumulative error entirely overwrites the trained parameters (Huang et al., 2026).
Laboratory apparatus demonstrating how a single towering activation outlier forces the surrounding signals to zero during absmax quantization.
🔍 Fact Check:While post-training quantization is often assumed to degrade models gracefully, uncorrected 1-bit compression triggers absolute representational collapse. Empirical testing reveals that uncorrected 1-bit models suffer from catastrophic perplexity explosions exceeding 1⁰⁸ on standard benchmarks like WikiText-2, rendering them completely inoperative (Huang et al., 2026).
To salvage post-training methods, researchers historically turned to mixed-precision workarounds like the LLM.int8() framework (Dettmers et al., 2022). This approach works by decomposing the matrix multiplication path (Dettmers et al., 2022). It programmatically isolates the outlier channels into a high-precision FP16 computational path while quantizing the remaining 99.9% of normal features to INT8 (Dettmers et al., 2022). While this dual-pathway decomposition successfully preserves model accuracy, it introduces severe physical trade-offs (Dettmers et al., 2022; Xiao et al., 2023). Modern graphic processors are designed to execute massive, highly uniform, dense matrix operations (Dettmers et al., 2022). Splitting a single layer into a “dense-and-sparse” pipeline creates irregular memory access patterns, complex register packing overheads, and severe control-flow pipeline divergence (Dettmers et al., 2022; Xiao et al., 2023). This physical bottleneck shift ultimately defeats the latency and throughput benefits that quantization was meant to deliver in the first place (Dettmers et al., 2022).
If late-stage model slicing leads to computational dead ends, the alternative is a philosophical shift (Ma et al., 2024). We must stop treating hardware and software as separate entities and embrace native co-design (Huang et al., 2026; Ma et al., 2024). Instead of squeezing a pre-trained model until it breaks, we must train models to exist natively in low-bit environments (Ma et al., 2024). This paradigm is best exemplified by the move toward native ternary architectures (Ma et al., 2024).
[ Native 1.58-Bit Ternary Weights ] │ ┌────────────────────────┴────────────────────────┐ ▼ ▼[ BitNet b1.58 Paradigm ] [ Hardware Alignment ] - Weight Set: {-1, 0, 1} - Standard register mismatch - log₂(3) ≈ 1.58 bits - Unaligned packing stalls - "0" state acts as native prune - Solution: Sherry 3:4 Sparsity - Eliminates FP multiplication - Result: 1.25-bit hardware alignment
At the forefront of this shift is the BitNet b1.58 architecture (Ma et al., 2024). BitNet replaces standard full-precision linear layers with specialized BitLinear layers during pre-training (Ma et al., 2024). Under this regime, every single parameter within the weight matrices is constrained to the ternary set {-1, 0, 1} (Ma et al., 2024). Because there are only three possible states, representing each weight requires exactly log₂(3) ≈ 1.58 bits of digital storage (Ma et al., 2024).
The inclusion of the 0 state is a structurally profound breakthrough (Ma et al., 2024). In a pure 1-bit binary system, a weight must be either active positive or active negative, leaving no room for subtle silence (Dettmers et al., 2022; Ma et al., 2024). The ternary ‘0’ state serves as a native, dynamic pruning mechanism, allowing the model to naturally filter out background noise and focus its representation capacity purely on salient features (Ma et al., 2024).
“Do not squeeze complexity into silence; train silence to hold complexity.”— Mohit Sewak, Ph.D.
Moreover, multiplying any activation by {-1, 0, 1} completely bypasses the need for power-hungry floating-point multiplications (Ma et al., 2024; Wang et al., 2025). The processor’s Arithmetic Logic Unit (ALU) can execute the entire matrix multiplication using only simple addition and subtraction operations (Ma et al., 2024; Wang et al., 2025). Highly optimized custom runtimes like bitnet.cpp allow a 100-billion parameter model to execute at 5–7 tokens per second entirely on consumer-grade CPUs (Wang et al., 2025). Furthermore, native ternary training is not a compromise (Ma et al., 2024). At scale, models like the BitNet b1.58 2B4T, trained from scratch on 4 trillion tokens, match the exact perplexity and zero-shot reasoning capabilities of state-of-the-art full-precision baselines (Ma et al., 2024).
However, pushing boundaries below the 1.58-bit limit introduces a physical register alignment crisis (Huang et al., 2026). Commercial hardware is designed to operate in strict power-of-two boundaries: 2-bit, 4-bit, or 8-bit registers (Huang et al., 2026). Attempting to pack an unaligned 1.58-bit weight into these registers leads to immediate memory inefficiencies (Huang et al., 2026). If you use standard 2-bit packing, you waste 0.42 bits of valuable memory bandwidth per parameter (Huang et al., 2026). If you attempt unaligned 1.67-bit packing, the computational pipeline stalls as the processor spends precious clock cycles performing complex bit-shifting operations just to unpack the weights (Huang et al., 2026).
To bypass this alignment bottleneck, architects must pair quantization with structural sparsity (Dong et al., 2025; Huang et al., 2026). The Structured Binarized LLM (STBLLM) framework uses N:M structural sparsity guided by a Standardized Importance (SI) metric to achieve sub-1-bit widths without causing binarization failure (Dong et al., 2025).
Physical toggle switchboard representing native ternary weights and the Sherry 3:4 sparsity pattern for 1.25-bit hardware alignment.
Even more elegantly, the Sherry framework applies a fine-grained 3:4 sparsity pattern to ternary networks (Huang et al., 2026). For every block of four weights, the Sherry framework forces the least important weight to zero, storing only the three active ternary weights in exactly 5 bits of memory (Huang et al., 2026). Let’s look at the math: packing 4 weights into 5 bits yields an overall width of:
5 ÷ 4 = 1.25 bits
This structural adjustment perfectly restores power-of-two memory alignment on commodity hardware (Huang et al., 2026). The physical payoff is massive, delivering up to a 10% decoding speedup and 25% total memory bandwidth savings on standard CPUs without any loss in model accuracy (Huang et al., 2026).
[Original Ternary Block] ──► [Sherry 3:4 Sparsity] ──► [5-Bit Register Packing] W1, W2, W3, W4 (Ternary) W1, W2, W3, 0 (Sparse) Packed as exactly 1.25 bits/weight
For model architects, the actionable takeaway here is clear (Huang et al., 2026). Do not try to salvage a pre-trained model with late-stage sub-4-bit post-training quantization (Ma et al., 2024). Instead, invest early in Quantization-Aware Training (QAT) pipelines that employ native ternary layers (Huang et al., 2026; Ma et al., 2024). During training, use Straight-Through Estimators (STE) to guide discrete forward passes, and implement heterogeneous gradient injection modules, like Sherry’s Arenas annealing system, to prevent latent weights from becoming trapped in sub-optimal regions (Huang et al., 2026).
💡 ProTip:To immunize ternary QAT pipelines against weight trapping under Straight-Through Estimators, avoid symmetric quantization of weights and activations at epoch zero. Force activations to initialize at 4-bit precision to maintain gradient variance, only progressively annealing them down to lower bit-widths once the weight distribution stabilizes into tri-modal clusters.
While native ternary weights solve half of the memory bottleneck, the real boss of inference latency is the activation space (Liu et al., 2024; Wang et al., 2024). Autoregressive decoding is a highly dynamic process (Xiao et al., 2023). If we compress the model’s weights to 1.58 bits but are forced to retain 8-bit activations (W1A8) to prevent representational collapse, we cannot utilize fast, low-power INT4 tensor core math pipelines (Ashkboos et al., 2024; Wang et al., 2025). The model is still forced to stream high-precision activations, limiting our computational throughput (Dettmers et al., 2022; Xiao et al., 2023). Pushing activations directly to 4-bit (W1A4), however, has historically resulted in immediate accuracy collapse (Ashkboos et al., 2024; Wang et al., 2024).
To understand why, think of self-attention as a crowded cocktail party (Liu et al., 2024). In a standard room, most people talk at a reasonable volume (Dettmers et al., 2022). But in a deep transformer, certain dynamic activation dimensions act like a guest screaming at the top of their lungs (Dettmers et al., 2022). If we try to quantize the room’s volume using a uniform 4-bit scale, we must set the maximum range to accommodate the screamer (Dettmers et al., 2022). As a result, every other normal conversation is rounded down to absolute silence (Dettmers et al., 2022; Xiao et al., 2023).
The definitive mathematical solution to this problem is non-destructive orthogonal rotation, specifically utilizing the Walsh-Hadamard Transform (WHT) (Ashkboos et al., 2024; Wang et al., 2025). Implemented in frameworks like QuaRot and operationalized in BitNet v2 via the “H-BitLinear” module, this transform acts as a spatial smoother (Ashkboos et al., 2024; Wang et al., 2025).
A physical wave-smoothing machine demonstrating the mathematical redistribution of spiky outliers into a uniform wave using a rotating Walsh-Hadamard disc array.
A Hadamard matrix is an explicitly constructed square matrix consisting entirely of +1 and -1 values, where every row is perfectly orthogonal to every other row (Ashkboos et al., 2024). By multiplying the activation matrix by a normalized Hadamard matrix, we rotate the coordinate space of the activations (Ashkboos et al., 2024; Wang et al., 2025). Let X be the activation matrix and H be the normalized Walsh-Hadamard matrix (Ashkboos et al., 2024). The transformed activations Y are calculated as:
Y = X × (H ÷ √N)
Where N represents the dimensionality of the hidden state (Ashkboos et al., 2024). Because the rotation is orthogonal, it is mathematically reversible and preserves the inner products of the vectors, ensuring no information is lost (Ashkboos et al., 2024).
However, the physical effect on the distribution of activations is profound (Ashkboos et al., 2024; Wang et al., 2025). The rotation projects the spiky, concentrated energy of the single outlier dimension across all N dimensions (Ashkboos et al., 2024; Wang et al., 2025). This converts a highly skewed, heavy-tailed distribution into a uniform, Gaussian-like shape (Ashkboos et al., 2024; Wang et al., 2025). The screaming guest is mathematically distributed into a harmonious chorus (Wang et al., 2025). The rotated activations now fit cleanly into a tight, uniform 4-bit integer grid without squashing normal features (Ashkboos et al., 2024; Wang et al., 2025).
[Spiky Activations X] ──► [Walsh-Hadamard Rotation] ──► [Smooth Activations Y] - Outliers: 10⁵ Y = X × (H ÷ √N) - Uniformly distributed - Normal: 0.1 - Fits perfectly in 4-bit grid
The computational elegance of this approach lies in its efficiency (Ashkboos et al., 2024; Wang et al., 2025). Because the Walsh-Hadamard Transform can be executed in O(N log N) operations using the Fast Walsh-Hadamard algorithm, it bypasses costly matrix multiplications entirely (Ashkboos et al., 2024; Wang et al., 2025). It introduces near-zero latency overhead during inference (Ashkboos et al., 2024; Wang et al., 2025). To ensure compatibility with architectures like Qwen, which use non-power-of-two embedding sizes, modern compilers utilize the Paley algorithm to construct appropriate orthogonal matrices (Ashkboos et al., 2024). By incorporating H-BitLinear modules, BitNet v2 successfully cracks the W1A4 barrier, enabling native 4-bit activation quantization with zero observable accuracy loss (Wang et al., 2025).
If rotational transformation is not an option, architects can utilize component-specific hybrid sparsification, as demonstrated by the BitNet a4.8 architecture (Wang et al., 2024). This framework acknowledges that different layers have different sensitivities to precision loss (Wang et al., 2024). It applies aggressive 4-bit quantization to the inputs of stable layers like Multi-Head Attention, but maintains an 8-bit pathway for the highly sensitive intermediate feed-forward states (Wang et al., 2024). To offset the memory footprint of the 8-bit path, BitNet a4.8 employs a squared ReLU gating mechanism that applies tight Top-K sparsification (Wang et al., 2024). At runtime, only 55% of the model parameters are activated (Wang et al., 2024). This hybrid configuration matches the throughput of a native W1A4 model while executing up to 4x faster than standard full-precision baselines (Wang et al., 2024).
As large language models radically scale their context windows from thousands to millions of tokens, the Key-Value (KV) cache quickly overtakes the model parameters as the dominant memory bottleneck during serving (Liu et al., 2024; Su et al., 2026; Zhang et al., 2024). The KV cache operates as a streaming data structure, incrementally storing keys and values for all past tokens to prevent costly re-computation (Liu et al., 2024). For a 175-billion parameter model processing a large batch size, the KV cache footprint can easily exceed 1.2 Terabytes, completely overwhelming on-chip SRAM and forcing high-latency off-chip memory swaps (Liu et al., 2024).
[ KV Cache Compression ] │ ┌─────────────────────────┴─────────────────────────┐ ▼ ▼[ Structural Asymmetry (KIVI) ] [ Token Norm Imbalance (OScaR) ] - Keys: Highly channel-wise outliers - Wild vector L₂ norm variance - Values: Token-wise outliers - Resolution: Canalized Rotation - Solution: Asymmetric 2-bit quantization followed by Omni-Token Scaling
Quantizing the KV cache to extreme low-bits (such as 2-bit or sub-1-bit) presents unique challenges because Key and Value matrices exhibit completely different topological behaviors (Liu et al., 2024; Su et al., 2026). Key matrices exhibit persistent, channel-wise activation outliers that span across the entire sequence dimension (Liu et al., 2024). Value matrices, on the other hand, function as dynamic attention mixing matrices and lack channel-wise outliers entirely; their variations are stochastically tied to individual tokens (Liu et al., 2024).
A physical architectural model illustrating asymmetric KV cache compression with dual-pathway per-channel key clamping and per-token value adjustment.
💡 ProTip:When compressing Key-Value caches for ultra-long context windows, never use uniform sequence-level quantization. Instead, leverage asymmetric quantization — group and scale Key matrices strictly per-channel to isolate static outliers, but scale Value matrices strictly per-token to accommodate dynamic runtime activations (Liu et al., 2024).
Applying uniform, symmetric quantization across both matrices destroys model performance (Liu et al., 2024). The KIVI framework (Liu et al., 2024) solves this structural asymmetry by introducing a tuning-free, asymmetric 2-bit quantization algorithm (Liu et al., 2024). KIVI quantizes the Key cache strictly per-channel, isolating channel-wise outliers to their specific dimensions (Liu et al., 2024). Conversely, it quantizes the Value cache strictly per-token (Liu et al., 2024). When combined with a tiny FP16 sliding window for the most recent tokens, this hardware-friendly algorithm achieves near-lossless 2-bit KV cache quantization (Liu et al., 2024). This allows systems to run up to 4x larger batch sizes and yields a 3.47x higher inference throughput on standard hardware (Liu et al., 2024).
However, pushing past the 2-bit barrier reveals a deeper topological flaw: Token Norm Imbalance (TNI) (Su et al., 2026). Across a long context window, the vector L₂ norm of individual token embeddings varies wildly (Su et al., 2026). Heavily attended tokens (such as attention sinks) possess massive activation energy, while normal tokens have negligible norms (Su et al., 2026). Under standard per-channel quantization, a shared scaling parameter is calculated across the sequence dimension (Su et al., 2026). If a channel contains both high-norm and low-norm tokens, the shared parameter must expand to accommodate the massive variance (Su et al., 2026). This systematically amplifies reconstruction errors on the low-norm tokens, causing complete quantization fidelity collapse (Su et al., 2026).
To resolve Token Norm Imbalance without relying on slow, complex pipelines, the OScaR (Omni-Scaled Canalized Rotation) framework executes a lightweight, dual-stage pipeline (Su et al., 2026):
By combining Canalized Rotation with Omni-Token Scaling, OScaR achieves lossless 2-bit (INT2) KV cache quantization (Su et al., 2026). It delivers a massive 5.3x reduction in memory footprint and up to a 3.0x decoding speedup compared to standard BF16 FlashDecoding-v2 baselines (Su et al., 2026).
🔍 Fact Check:While standard 16-bit Key-Value caches for massive transformer configurations easily balloon past 1.2 Terabytes, the OScaR framework leverages a dual-stage pipeline of Canalized Rotation and Omni-Token Scaling to enable lossless 2-bit (INT2) KV cache quantization, driving a 5.3x reduction in memory footprint and up to a 3.0x decoding speedup (Su et al., 2026).
[Raw Key/Value Tensors] ──► [Canalized Rotation] ──► [Omni-Token Scaling] ──► [Lossless INT2 KV Cache] - Token Norm Imbalance - Outlier smoothing - Balanced L₂ norms - 5.3x memory footprint reduction
To push beyond the 1-bit barrier, the Coupled Quantization (CQ) framework abandons the assumption that embedding channels are strictly independent (Zhang et al., 2024). CQ actively exploits the deep spatial correlation and mutual information shared between adjacent activation channels (Zhang et al., 2024). Instead of quantizing individual channels, CQ groups contiguous channels (e.g., blocks of 16) into jointly encoded multi-dimensional vectors (Zhang et al., 2024).
To preserve maximum fidelity, the multi-channel centroids are learned offline using Fisher-guided learning (Zhang et al., 2024). This method uses a second-order Fisher Information matrix to prioritize and preserve high-impact activations during centroid construction (Zhang et al., 2024). The CQ-16c12b configuration uses a 12-bit codebook index to represent 16 coupled channels, achieving an astounding average of 0.81 bits per activation (Zhang et al., 2024). When deployed with a tiny high-precision sliding window, Coupled Quantization fully preserves model quality while supporting up to 15x larger batch sizes than standard FP16 baselines, unlocking massive data center throughput gains (Zhang et al., 2024).
A physical installation of glass capillaries with glowing light pulses demonstrating native 1-bit neuromorphic binary event gating.
As traditional quantization and ternary weight frameworks reach their theoretical minimums, vanguard research is shifting toward the ultimate limit of inference: purely 1-bit binary activations combined with 1-bit weights (Zhu et al., 2023). Standard large language models require dense, continuous input activations to propagate gradients (Zhu et al., 2023). Forcing both weights and activations to 1-bit simultaneously typically results in immediate accuracy collapse (Zhu et al., 2023).
To overcome this, researchers are turning to Spiking Neural Networks (SNNs) (Zhu et al., 2023). Event-driven architectures like SpikeGPT and Dual-Path SparseTCAM replace dense, continuous activations with discrete binary spikes {0, 1} and Leaky Integrate-and-Fire temporal dynamics (Zhu et al., 2023). Instead of executing complex floating-point matrix math, SpikeGPT treats binary spikes as event-driven logic gates (Zhu et al., 2023). While these architectures currently incur a slight quality penalty compared to dense transformers, they demonstrate that high global activation sparsity is highly compatible with language modeling (Zhu et al., 2023). Moving from dense mathematical operations to binary event gating paves the way for ultra-low latency, power-efficient language models run entirely on specialized neuromorphic silicon (Zhu et al., 2023).
To navigate this rapidly evolving landscape, we must contrast the traditional post-training methods with next-generation co-designed low-bit frameworks:
Feature Dimension Traditional Post-Training Quantization (PTQ) Next-Gen Co-Designed Inference Weight Representation Lossy 4-bit or 8-bit integer maps (Dettmers et al., 2022; Xiao et al., 2023) Native 1.58-bit ternary weight layers ({-1, 0, 1}) (Ma et al., 2024) Outlier Mitigation Heavy runtime mixed-precision pipelines (Dettmers et al., 2022; Xiao et al., 2023) Online Walsh-Hadamard rotational smoothing (Ashkboos et al., 2024; Wang et al., 2025) KV Cache Compression Linear downscaling or aggressive truncation (Liu et al., 2024) Asymmetric, norm-balanced KIVI / OScaR pipelines (Liu et al., 2024; Su et al., 2026) Core ALU Execution Complex floating-point matrix multiplications (Ma et al., 2024; Wang et al., 2025) High-efficiency addition/subtraction pipelines (Ma et al., 2024; Wang et al., 2025)
The pursuit of extreme model compression has officially moved beyond simple post-training hacks (Ma et al., 2024). The mathematical barriers that once made sub-2-bit inference impossible — the catastrophic loss of parameter expressiveness and the tyranny of emergent activation outliers — are being systematically dismantled through elegant, co-designed architecture (Dettmers et al., 2022; Ma et al., 2024; Xiao et al., 2023).
For machine learning engineers, infrastructure architects, and technology leaders, the message is clear (Dettmers et al., 2022; Ma et al., 2024). Stop trying to slice, dice, and salvage pre-trained models after the fact (Ma et al., 2024). The path to cost-effective, high-throughput, edge-capable AI requires investing early in native low-bit training (Huang et al., 2026; Ma et al., 2024). By embracing native ternary weights, online rotational transformations, and asymmetric, norm-balanced KV cache compression, we can deploy state-of-the-art models at a fraction of the computational and financial footprint (Ashkboos et al., 2024; Liu et al., 2024; Ma et al., 2024; Su et al., 2026; Wang et al., 2025). Let us put down the post-training woodchipers and start building our models to fit the silicon from the very first epoch (Huang et al., 2026; Ma et al., 2024).
Dettmers, T., Lewis, M., Belkada, Y., & Zettlemoyer, L. (2022). LLM.int8(): 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 35, 30318–30332. https://doi.org/10.48550/arXiv.2208.07339
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., & Han, S. (2023). SmoothQuant: Accurate and efficient post-training quantization for large language models. Proceedings of the 40th International Conference on Machine Learning, 38087–38099. https://doi.org/10.48550/arXiv.2211.10438
Dong, P., Li, L., Zhong, Y., Du, D., Fan, R., Chen, Y., Tang, Z., Wang, Q., Xue, W., Guo, Y., & Chu, X. (2025). STBLLM: Breaking the 1-bit barrier with structured binary LLMs. International Conference on Learning Representations (ICLR 2025). https://doi.org/10.48550/arXiv.2408.01803
Huang, H., Wu, D., Hu, Q., Yu, G., Yang, J., Zhu, J., Liu, X., & Wu, D. (2026). Sherry: Hardware-efficient 1.25-bit ternary quantization via fine-grained sparsification. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 11186–11204. https://doi.org/10.18653/v1/2026.acl-long.513
Ma, S., Wang, H., Ma, L., Wang, L., Wang, W., Huang, S., Dong, L., Wang, R., Xue, J., & Wei, F. (2024). The era of 1-bit LLMs: All large language models are in 1.58 bits. arXiv preprint arXiv:2402.17764. https://doi.org/10.48550/arXiv.2402.17764
Wang, J., Zhou, H., Song, T., Cao, S., Xia, Y., Cao, T., Wei, J., Ma, S., Wang, H., & Wei, F. (2025). Bitnet.cpp: Efficient edge inference for ternary LLMs. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9305–9322. https://doi.org/10.18653/v1/2025.acl-long.457
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B., Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., & Hensman, J. (2024). QuaRot: Outlier-free 4-bit inference in rotated LLMs. Advances in Neural Information Processing Systems, 37, 10023–10045. https://doi.org/10.48550/arXiv.2403.17319
Wang, H., Ma, S., & Wei, F. (2024). BitNet a4.8: 4-bit activations for 1-bit LLMs. arXiv preprint arXiv:2411.04965. https://doi.org/10.48550/arXiv.2411.04965
Wang, H., Ma, S., & Wei, F. (2025). BitNet v2: Native 4-bit activations with Hadamard transformation for 1-bit LLMs. arXiv preprint arXiv:2504.18415. https://doi.org/10.48550/arXiv.2504.18415
Liu, Z., Yuan, J., Jin, H., Zhong, S., Xu, Z., Braverman, V., Chen, B., & Hu, X. (2024). KIVI: A tuning-free asymmetric 2-bit quantization for KV cache. Proceedings of the 41st International Conference on Machine Learning (ICML 2024), 32332–32350. https://doi.org/10.48550/arXiv.2402.02750
Su, Z., Yang, R., Zhang, C., Liu, Y., Zhang, Y., Wu, W., Xiong, J., Du, D., Zhuang, X., Qian, Y., Xie, Y., Wu, Y.-C., Yang, H., & Wong, N. (2026). OScaR: The Occam’s razor for extreme KV cache quantization in LLMs and beyond. arXiv preprint arXiv:2605.14185. https://doi.org/10.48550/arXiv.2605.14185
Zhang, T., Yi, J., Xu, Z., & Shrivastava, A. (2024). KV cache is 1 bit per channel: Efficient large language model inference with coupled quantization. Advances in Neural Information Processing Systems, 37, 21456–21478. https://doi.org/10.48550/arXiv.2405.03917
Zhu, R.-J., Zhao, Q., Li, G., & Eshraghian, J. K. (2023). SpikeGPT: Generative pre-trained language model with spiking neural networks. Transactions on Machine Learning Research, 2023. https://doi.org/10.48550/arXiv.2302.13939
Disclaimer: The views and opinions expressed in this article are personal and do not necessarily reflect the official policy or position of any associated agencies, organizations, or the India AI Mission. AI assistance was utilized in the research, drafting, and ideation of this article. Licensed under CC BY-ND 4.0.
Why Post-Training Compression Is a Losing LLM Battle was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.