Model Compression Beyond Quantization A tutorial on model compression beyond quantization details how pruning, structured pruning and knowledge distillation can shrink models for cheaper inference, illustrating magnitude pruning with a PyTorch example that masks a 1024x1024 weight matrix at 50% sparsity and converts it to compressed sparse row (CSR) format. The post notes that sparse representations save space only when metadata costs less than the removed data, and speed up execution only when the matrix shape, sparsity level, dtype and hardware have an efficient sparse kernel. Structured pruning removes complete units such as attention heads and MLP channels, while distillation trains a smaller student model to reproduce a teacher's behavior. Quantization changes how model values are represented. Other compression methods reduce the number of values or the size of model structures. Pruning removes weights or structures, while knowledge distillation trains a smaller student to reproduce a teacher’s behavior. Serving a smaller model is often an effective inference optimization. It reads fewer weights and may use less cache, depending on its layer count, KV heads, and head dimension. It may also allow larger batches or cheaper hardware. The difficult part is preserving the behavior the application needs and ensuring that the compressed structure is supported efficiently by the runtime. In this post, you will learn about: - How pruning removes weights and when sparsity can improve execution - How structured pruning changes model shapes and compute - How distillation trains a smaller model for serving Let’s get started. Overview This post is divided into three parts; they are: - Pruning and Sparsity - Structured Pruning - Knowledge Distillation Pruning and Sparsity Pruning sets selected model parameters to zero or removes larger structures. Magnitude pruning starts with a simple assumption: weights with small absolute values may have less effect on the output. python Apply magnitude pruning to a linear layer and compare its output. import torch torch.manual seed 0 def magnitude mask weight, sparsity : assert 0.0 <= sparsity < 1.0 if sparsity == 0.0: return torch.ones like weight, dtype=torch.bool threshold = torch.quantile weight.abs .float , sparsity return weight.abs threshold weight = torch.randn 1024, 1024 mask = magnitude mask weight, sparsity=0.5 pruned weight = weight mask print "remaining fraction:", mask.float .mean .item Convert the pruned matrix to compressed sparse row CSR format. sparse weight = pruned weight.to sparse csr Multiply a dense batch by the sparse weight matrix. x = torch.randn 8, 1024 sparse output = torch.sparse.mm sparse weight, x.T .T dense output = x @ pruned weight.T torch.testing.assert close sparse output, dense output, rtol=1e-4, atol=2e-4 1234567891011121314151617181920212223 import torch torch.manual seed 0 def magnitude mask weight, sparsity : assert 0.0 <= sparsity < 1.0 if sparsity == 0.0: return torch.ones like weight, dtype=torch.bool threshold = torch.quantile weight.abs .float , sparsity return weight.abs threshold weight = torch.randn 1024, 1024 mask = magnitude mask weight, sparsity=0.5 pruned weight = weight maskprint "remaining fraction:", mask.float .mean .item Convert the pruned matrix to compressed sparse row CSR format.sparse weight = pruned weight.to sparse csr Multiply a dense batch by the sparse weight matrix.x = torch.randn 8, 1024 sparse output = torch.sparse.mm sparse weight, x.T .Tdense output = x @ pruned weight.Ttorch.testing.assert close sparse output, dense output, rtol=1e-4, atol=2e-4 Sparse and dense matrix multiplication can accumulate terms in a different order, so the script compares their outputs with a numerical tolerance. The sparsity argument represents the fraction of weights intended for removal. Setting sparsity=0.0 retains all weights. The upper bound excludes sparsity=1.0 , which would remove every weight and make the layer unusable. After pruning, zeros can appear anywhere in pruned weight , but that tensor is still stored densely. Calling to sparse csr creates a compressed sparse row tensor that stores the nonzero values together with row and column metadata. The function torch.sparse.mm then multiplies that sparse matrix by a dense matrix. A sparse representation saves space only when its metadata costs less than the removed data, and it improves speed only when the matrix shape, sparsity level, dtype, and hardware have an efficient sparse kernel. Structured Pruning Structured pruning removes complete units such as attention heads, MLP channels, layers, or blocks. The result can often be represented as a smaller dense model, which makes it easier to accelerate across hardware. A transformer MLP uses one projection to expand each hidden vector into an intermediate representation and another to project it back to the model’s hidden size: $$ \mathbf{y} = \mathbf{W} {\text{down}} \phi \mathbf{W} {\text{up}} \mathbf{x} $$ Each intermediate channel is produced by one row of $\mathbf{W} {\text{up}}$ and consumed by the matching column of $\mathbf{W} {\text{down}}$. Removing a row from the up projection therefore requires removing the corresponding column from the down projection. The following small example removes intermediate channels according to the norms of the first projection’s rows: python Remove low-norm intermediate channels from a two-layer projection. import torch import torch.nn as nn def shrink mlp up proj, down proj, keep channels : if not 1 <= keep channels <= up proj.out features: raise ValueError "keep channels must be within the MLP width" if up proj.out features = down proj.in features: raise ValueError "projection widths must match" scores = up proj.weight.norm dim=1 keep = scores.topk keep channels .indices.sort .values smaller up = nn.Linear up proj.in features, keep channels, bias=up proj.bias is not None, .to device=up proj.weight.device, dtype=up proj.weight.dtype smaller down = nn.Linear keep channels, down proj.out features, bias=down proj.bias is not None, .to device=down proj.weight.device, dtype=down proj.weight.dtype with torch.no grad : smaller up.weight.copy up proj.weight keep smaller down.weight.copy down proj.weight :, keep if up proj.bias is not None: smaller up.bias.copy up proj.bias keep if down proj.bias is not None: smaller down.bias.copy down proj.bias return smaller up, smaller down up = nn.Linear 16, 32 down = nn.Linear 32, 16 smaller up, smaller down = shrink mlp up, down, keep channels=16 print smaller down torch.relu smaller up torch.randn 2, 16 .shape 123456789101112131415161718192021222324252627282930313233343536 import torchimport torch.nn as nn def shrink mlp up proj, down proj, keep channels : if not 1 <= keep channels <= up proj.out features: raise ValueError "keep channels must be within the MLP width" if up proj.out features = down proj.in features: raise ValueError "projection widths must match" scores = up proj.weight.norm dim=1 keep = scores.topk keep channels .indices.sort .values smaller up = nn.Linear up proj.in features, keep channels, bias=up proj.bias is not None, .to device=up proj.weight.device, dtype=up proj.weight.dtype smaller down = nn.Linear keep channels, down proj.out features, bias=down proj.bias is not None, .to device=down proj.weight.device, dtype=down proj.weight.dtype with torch.no grad : smaller up.weight.copy up proj.weight keep smaller down.weight.copy down proj.weight :, keep if up proj.bias is not None: smaller up.bias.copy up proj.bias keep if down proj.bias is not None: smaller down.bias.copy down proj.bias return smaller up, smaller down up = nn.Linear 16, 32 down = nn.Linear 32, 16 smaller up, smaller down = shrink mlp up, down, keep channels=16 print smaller down torch.relu smaller up torch.randn 2, 16 .shape The parameter keep channels is the number of intermediate MLP channels retained after pruning; it is a count, not a sparsity fraction. It must be an integer from 1 through up proj.out features , and the original projections must satisfy up proj.out features == down proj.in features . The code ranks the rows of up proj.weight by their norms and keeps the keep channels highest-scoring rows. It then keeps the matching columns of down proj.weight , so the reduced output of smaller up remains a valid input to smaller down . For example, setting keep channels=2048 reduces an original intermediate width of 8192 to 2048 channels. This code preserves compatible tensor shapes but does not determine whether the chosen channels are truly unimportant. A real pruning method evaluates layer outputs, interactions, and recovery fine-tuning. Removing attention heads requires similar coordinated edits to query, key, value, and output projections. With GQA, query heads share KV heads, so arbitrary head removal can break grouping. Removing transformer layers changes residual depth and may require learning mappings between the retained layers. These changes require coordinated edits and quality checks. Structured pruning can reduce parameter count and dense compute, but awkward hidden sizes may perform worse than slightly larger hardware-friendly sizes. Align channel counts with the tile requirements of the target matrix multiplication kernels. Knowledge Distillation A smaller model can run faster and use less memory. Knowledge distillation trains a smaller student model using signals from a larger teacher. The student may learn from the teacher’s output distribution not just the highest-scoring token , intermediate representations, generated data, or a combination of them. For logits $\mathbf{z} t$ and $\mathbf{z} s$ from teacher and student, a common distillation loss function uses a temperature $T$: $$ \mathcal{L} {KD} = T^2\operatorname{KL}\Big \operatorname{softmax}\big \frac{\mathbf{z} t}{T}\big \;\big\Vert \;\operatorname{softmax}\big \frac{\mathbf{z} s}{T}\big \Big $$ Here, $\operatorname{KL} \mathbf{P}\Vert \mathbf{Q} $ is the Kullback-Leibler divergence from the teacher distribution $\mathbf{P}$ to the student distribution $\mathbf{Q}$; it is zero when they match and grows as their probabilities differ. The corresponding PyTorch code is: Compute a temperature-scaled distillation loss from teacher and student logits. import torch import torch.nn.functional as F def distillation loss student logits, teacher logits, temperature=2.0 : if temperature <= 0: raise ValueError "temperature must be positive" teacher probs = F.softmax teacher logits / temperature, dim=-1 student log probs = F.log softmax student logits / temperature, dim=-1 return F.kl div student log probs, teacher probs, reduction="none" .sum dim=-1 .mean temperature 2 student logits = torch.tensor 1.0, 0.0 teacher logits = torch.tensor 0.0, 1.0 print "distillation loss:", distillation loss student logits, teacher logits .item 123456789101112131415161718 import torchimport torch.nn.functional as F def distillation loss student logits, teacher logits, temperature=2.0 : if temperature <= 0: raise ValueError "temperature must be positive" teacher probs = F.softmax teacher logits / temperature, dim=-1 student log probs = F.log softmax student logits / temperature, dim=-1 return F.kl div student log probs, teacher probs, reduction="none" .sum dim=-1 .mean temperature 2 student logits = torch.tensor 1.0, 0.0 teacher logits = torch.tensor 0.0, 1.0 print "distillation loss:", distillation loss student logits, teacher logits .item For padded batches, replace .mean with a masked average over valid token positions. This loss assumes teacher and student logits refer to the same vocabulary and token positions. The teacher runs without gradients, but producing its targets still costs time. The next example defines the same loss so it can run independently. The following local example uses two small, randomly initialized GPT-2 style models. It demonstrates one training step; it does not produce a useful language model. A real distillation experiment needs a pretrained teacher and representative text. Run one student training step with a temperature-scaled distillation loss. import torch import torch.nn.functional as F from transformers import GPT2Config, GPT2LMHeadModel def distillation loss student logits, teacher logits, temperature=2.0 : if temperature <= 0: raise ValueError "temperature must be positive" teacher probs = F.softmax teacher logits / temperature, dim=-1 student log probs = F.log softmax student logits / temperature, dim=-1 return F.kl div student log probs, teacher probs, reduction="none" .sum dim=-1 .mean temperature 2 torch.manual seed 0 vocab size = 128 teacher = GPT2LMHeadModel GPT2Config vocab size=vocab size, n layer=4, n head=4, n embd=64, .eval student = GPT2LMHeadModel GPT2Config vocab size=vocab size, n layer=2, n head=4, n embd=64, .train optimizer = torch.optim.AdamW student.parameters , lr=1e-3 tokens = torch.randint 0, vocab size, 4, 24 with torch.no grad : teacher logits = teacher tokens .logits :, :-1, : student logits = student tokens .logits :, :-1, : hard loss = F.cross entropy student logits.reshape -1, vocab size , tokens :, 1: .reshape -1 soft loss = distillation loss student logits, teacher logits loss = 0.5 hard loss + 0.5 soft loss optimizer.zero grad loss.backward optimizer.step print "training loss:", loss.item 123456789101112131415161718192021222324252627282930313233343536373839 import torchimport torch.nn.functional as Ffrom transformers import GPT2Config, GPT2LMHeadModel def distillation loss student logits, teacher logits, temperature=2.0 : if temperature <= 0: raise ValueError "temperature must be positive" teacher probs = F.softmax teacher logits / temperature, dim=-1 student log probs = F.log softmax student logits / temperature, dim=-1 return F.kl div student log probs, teacher probs, reduction="none" .sum dim=-1 .mean temperature 2 torch.manual seed 0 vocab size = 128teacher = GPT2LMHeadModel GPT2Config vocab size=vocab size, n layer=4, n head=4, n embd=64, .eval student = GPT2LMHeadModel GPT2Config vocab size=vocab size, n layer=2, n head=4, n embd=64, .train optimizer = torch.optim.AdamW student.parameters , lr=1e-3 tokens = torch.randint 0, vocab size, 4, 24 with torch.no grad : teacher logits = teacher tokens .logits :, :-1, : student logits = student tokens .logits :, :-1, : hard loss = F.cross entropy student logits.reshape -1, vocab size , tokens :, 1: .reshape -1 soft loss = distillation loss student logits, teacher logits loss = 0.5 hard loss + 0.5 soft lossoptimizer.zero grad loss.backward optimizer.step print "training loss:", loss.item For a real comparison, train on a representative data split and report held-out task quality, parameter bytes, peak memory, prefill latency, and decode throughput for the same prompts and hardware. Pruning or distillation can change behavior even when parameter count falls substantially. Further Reading Below are some resources you may find useful: - Jonathan Frankle, Michael Carbin 2018 . The Lottery Ticket Hypothesis https://arxiv.org/abs/1803.03635 . This paper studies sparse subnetworks that can train effectively within larger networks. - Elias Frantar, Dan Alistarh 2023 . SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot https://arxiv.org/abs/2301.00774 . This paper presents a post-training pruning method for large language models. - Geoffrey Hinton, Oriol Vinyals, Jeff Dean 2015 . Distilling the Knowledge in a Neural Network https://arxiv.org/abs/1503.02531 . This paper introduces the commonly used soft-target distillation formulation. - Victor Sanh et al. 2019 . DistilBERT https://arxiv.org/abs/1910.01108 . This paper applies knowledge distillation while pretraining a smaller transformer. Summary In this post, you learned that compression can change model structure, not only number formats. Unstructured pruning needs sparse storage and kernels to improve runtime, while structured pruning can produce smaller dense shapes. Distillation transfers behavior into a smaller student model.