cd /news/machine-learning/model-compression-beyond-quantizatio… · home › topics › machine-learning › article
[ARTICLE · art-172810] src=machinelearningmastery.com ↗ pub= topic=machine-learning verified=true sentiment=· neutral

Model Compression Beyond Quantization

A tutorial on model compression beyond quantization details how pruning, structured pruning and knowledge distillation can shrink models for cheaper inference, illustrating magnitude pruning with a PyTorch example that masks a 1024x1024 weight matrix at 50% sparsity and converts it to compressed sparse row (CSR) format. The post notes that sparse representations save space only when metadata costs less than the removed data, and speed up execution only when the matrix shape, sparsity level, dtype and hardware have an efficient sparse kernel. Structured pruning removes complete units such as attention heads and MLP channels, while distillation trains a smaller student model to reproduce a teacher's behavior.

recommendation

Convert pruned weight matrices to CSR with to_sparse_csr() and verify sparse versus dense outputs within tolerance, but only rely on sparsity for speed when the shape, dtype and hardware have an efficient sparse kernel.

more in playbook: Inference and self-hosting
by read9 min views1 publishedNov 19, 2026
Model Compression Beyond Quantization
Image: source

Quantization changes how model values are represented. Other compression methods reduce the number of values or the size of model structures. Pruning removes weights or structures, while knowledge distillation trains a smaller student to reproduce a teacher’s behavior.

Serving a smaller model is often an effective inference optimization. It reads fewer weights and may use less cache, depending on its layer count, KV heads, and head dimension. It may also allow larger batches or cheaper hardware. The difficult part is preserving the behavior the application needs and ensuring that the compressed structure is supported efficiently by the runtime.

In this post, you will learn about:

  • How pruning removes weights and when sparsity can improve execution
  • How structured pruning changes model shapes and compute
  • How distillation trains a smaller model for serving

Let’s get started.

Overview #

This post is divided into three parts; they are:

  • Pruning and Sparsity
  • Structured Pruning
  • Knowledge Distillation

Pruning and Sparsity #

Pruning sets selected model parameters to zero or removes larger structures. Magnitude pruning starts with a simple assumption: weights with small absolute values may have less effect on the output.

Apply magnitude pruning to a linear layer and compare its output.

import torch

torch.manual_seed(0)

def magnitude_mask(weight, sparsity):
    assert 0.0 <= sparsity < 1.0
    if sparsity == 0.0:
        return torch.ones_like(weight, dtype=torch.bool)
    threshold = torch.quantile(weight.abs().float(), sparsity)
    return weight.abs() > threshold

weight = torch.randn(1024, 1024)
mask = magnitude_mask(weight, sparsity=0.5)
pruned_weight = weight * mask
print("remaining fraction:", mask.float().mean().item())

sparse_weight = pruned_weight.to_sparse_csr()
x = torch.randn(8, 1024)
sparse_output = torch.sparse.mm(sparse_weight, x.T).T
dense_output = x @ pruned_weight.T
torch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4)

1234567891011121314151617181920212223

import torch torch.manual_seed(0) def magnitude_mask(weight, sparsity):    assert 0.0 <= sparsity < 1.0    if sparsity == 0.0:        return torch.ones_like(weight, dtype=torch.bool)    threshold = torch.quantile(weight.abs().float(), sparsity)    return weight.abs() > threshold weight = torch.randn(1024, 1024)mask = magnitude_mask(weight, sparsity=0.5)pruned_weight = weight * maskprint("remaining fraction:", mask.float().mean().item()) # Convert the pruned matrix to compressed sparse row (CSR) format.sparse_weight = pruned_weight.to_sparse_csr()# Multiply a dense batch by the sparse weight matrix.x = torch.randn(8, 1024)sparse_output = torch.sparse.mm(sparse_weight, x.T).Tdense_output = x @ pruned_weight.Ttorch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4)

Sparse and dense matrix multiplication can accumulate terms in a different order, so the script compares their outputs with a numerical tolerance. The sparsity argument represents the fraction of weights intended for removal. Setting sparsity=0.0 retains all weights. The upper bound excludes sparsity=1.0, which would remove every weight and make the layer unusable. After pruning, zeros can appear anywhere in pruned_weight, but that tensor is still stored densely. Calling to_sparse_csr() creates a compressed sparse row tensor that stores the nonzero values together with row and column metadata. The function torch.sparse.mm() then multiplies that sparse matrix by a dense matrix. A sparse representation saves space only when its metadata costs less than the removed data, and it improves speed only when the matrix shape, sparsity level, dtype, and hardware have an efficient sparse kernel.

Structured Pruning #

Structured pruning removes complete units such as attention heads, MLP channels, layers, or blocks. The result can often be represented as a smaller dense model, which makes it easier to accelerate across hardware.

A transformer MLP uses one projection to expand each hidden vector into an intermediate representation and another to project it back to the model’s hidden size:

$$

\mathbf{y} = \mathbf{W}{\text{down}} \phi(\mathbf{W}{\text{up}} \mathbf{x})

$$

Each intermediate channel is produced by one row of $\mathbf{W}{\text{up}}$ and consumed by the matching column of $\mathbf{W}{\text{down}}$. Removing a row from the up projection therefore requires removing the corresponding column from the down projection. The following small example removes intermediate channels according to the norms of the first projection’s rows:

Remove low-norm intermediate channels from a two-layer projection.

import torch
import torch.nn as nn

def shrink_mlp(up_proj, down_proj, keep_channels):
    if not 1 <= keep_channels <= up_proj.out_features:
        raise ValueError("keep_channels must be within the MLP width")
    if up_proj.out_features != down_proj.in_features:
        raise ValueError("projection widths must match")
    scores = up_proj.weight.norm(dim=1)
    keep = scores.topk(keep_channels).indices.sort().values

    smaller_up = nn.Linear(
        up_proj.in_features,
        keep_channels,
        bias=up_proj.bias is not None,
    ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype)
    smaller_down = nn.Linear(
        keep_channels,
        down_proj.out_features,
        bias=down_proj.bias is not None,
    ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype)

    with torch.no_grad():
        smaller_up.weight.copy_(up_proj.weight[keep])
        smaller_down.weight.copy_(down_proj.weight[:, keep])
        if up_proj.bias is not None:
            smaller_up.bias.copy_(up_proj.bias[keep])
        if down_proj.bias is not None:
            smaller_down.bias.copy_(down_proj.bias)

    return smaller_up, smaller_down

up = nn.Linear(16, 32)
down = nn.Linear(32, 16)
smaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)
print(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape)

123456789101112131415161718192021222324252627282930313233343536

import torchimport torch.nn as nn def shrink_mlp(up_proj, down_proj, keep_channels):    if not 1 <= keep_channels <= up_proj.out_features:        raise ValueError("keep_channels must be within the MLP width")    if up_proj.out_features != down_proj.in_features:        raise ValueError("projection widths must match")    scores = up_proj.weight.norm(dim=1)    keep = scores.topk(keep_channels).indices.sort().values     smaller_up = nn.Linear(        up_proj.in_features,        keep_channels,        bias=up_proj.bias is not None,    ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype)    smaller_down = nn.Linear(        keep_channels,        down_proj.out_features,        bias=down_proj.bias is not None,    ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype)     with torch.no_grad():        smaller_up.weight.copy_(up_proj.weight[keep])        smaller_down.weight.copy_(down_proj.weight[:, keep])        if up_proj.bias is not None:            smaller_up.bias.copy_(up_proj.bias[keep])        if down_proj.bias is not None:            smaller_down.bias.copy_(down_proj.bias)     return smaller_up, smaller_down up = nn.Linear(16, 32)down = nn.Linear(32, 16)smaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)print(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape)

The parameter keep_channels is the number of intermediate MLP channels retained after pruning; it is a count, not a sparsity fraction. It must be an integer from 1 through up_proj.out_features, and the original projections must satisfy up_proj.out_features == down_proj.in_features. The code ranks the rows of up_proj.weight by their norms and keeps the keep_channels highest-scoring rows. It then keeps the matching columns of down_proj.weight, so the reduced output of smaller_up remains a valid input to smaller_down. For example, setting keep_channels=2048 reduces an original intermediate width of 8192 to 2048 channels.

This code preserves compatible tensor shapes but does not determine whether the chosen channels are truly unimportant. A real pruning method evaluates layer outputs, interactions, and recovery fine-tuning.

Removing attention heads requires similar coordinated edits to query, key, value, and output projections. With GQA, query heads share KV heads, so arbitrary head removal can break grouping. Removing transformer layers changes residual depth and may require learning mappings between the retained layers. These changes require coordinated edits and quality checks.

Structured pruning can reduce parameter count and dense compute, but awkward hidden sizes may perform worse than slightly larger hardware-friendly sizes. Align channel counts with the tile requirements of the target matrix multiplication kernels.

Knowledge Distillation #

A smaller model can run faster and use less memory. Knowledge distillation trains a smaller student model using signals from a larger teacher. The student may learn from the teacher’s output distribution (not just the highest-scoring token), intermediate representations, generated data, or a combination of them.

For logits $\mathbf{z}_t$ and $\mathbf{z}_s$ from teacher and student, a common distillation loss function uses a temperature $T$:

$$

\mathcal{L}_{KD} = T^2\operatorname{KL}\Big(

\operatorname{softmax}\big(\frac{\mathbf{z}_t}{T}\big);\big\Vert ;\operatorname{softmax}\big(\frac{\mathbf{z}_s}{T}\big)

\Big)

$$

Here, $\operatorname{KL}(\mathbf{P}\Vert \mathbf{Q})$ is the Kullback-Leibler divergence from the teacher distribution $\mathbf{P}$ to the student distribution $\mathbf{Q}$; it is zero when they match and grows as their probabilities differ. The corresponding PyTorch code is:

Compute a temperature-scaled distillation loss from teacher and student logits.

import torch
import torch.nn.functional as F

def distillation_loss(student_logits, teacher_logits, temperature=2.0):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)
    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)
    return (
        F.kl_div(student_log_probs, teacher_probs, reduction="none")
        .sum(dim=-1)
        .mean()
        * temperature**2
    )

student_logits = torch.tensor([[1.0, 0.0]])
teacher_logits = torch.tensor([[0.0, 1.0]])
print("distillation loss:", distillation_loss(student_logits, teacher_logits).item())

123456789101112131415161718

import torchimport torch.nn.functional as F def distillation_loss(student_logits, teacher_logits, temperature=2.0):    if temperature <= 0:        raise ValueError("temperature must be positive")    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)    return (        F.kl_div(student_log_probs, teacher_probs, reduction="none")        .sum(dim=-1)        .mean()        * temperature**2    ) student_logits = torch.tensor([[1.0, 0.0]])teacher_logits = torch.tensor([[0.0, 1.0]])print("distillation loss:", distillation_loss(student_logits, teacher_logits).item())

For padded batches, replace .mean() with a masked average over valid token positions. This loss assumes teacher and student logits refer to the same vocabulary and token positions. The teacher runs without gradients, but producing its targets still costs time. The next example defines the same loss so it can run independently.

The following local example uses two small, randomly initialized GPT-2 style models. It demonstrates one training step; it does not produce a useful language model. A real distillation experiment needs a pretrained teacher and representative text.

Run one student training step with a temperature-scaled distillation loss.

import torch
import torch.nn.functional as F
from transformers import GPT2Config, GPT2LMHeadModel

def distillation_loss(student_logits, teacher_logits, temperature=2.0):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)
    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)
    return (
        F.kl_div(student_log_probs, teacher_probs, reduction="none")
        .sum(dim=-1)
        .mean()
        * temperature**2
    )

torch.manual_seed(0)
vocab_size = 128
teacher = GPT2LMHeadModel(GPT2Config(
    vocab_size=vocab_size, n_layer=4, n_head=4, n_embd=64,
)).eval()
student = GPT2LMHeadModel(GPT2Config(
    vocab_size=vocab_size, n_layer=2, n_head=4, n_embd=64,
)).train()
optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)
tokens = torch.randint(0, vocab_size, (4, 24))

with torch.no_grad():
    teacher_logits = teacher(tokens).logits[:, :-1, :]
student_logits = student(tokens).logits[:, :-1, :]
hard_loss = F.cross_entropy(
    student_logits.reshape(-1, vocab_size), tokens[:, 1:].reshape(-1)
)
soft_loss = distillation_loss(student_logits, teacher_logits)
loss = 0.5 * hard_loss + 0.5 * soft_loss
optimizer.zero_grad()
loss.backward()
optimizer.step()
print("training loss:", loss.item())

123456789101112131415161718192021222324252627282930313233343536373839

import torchimport torch.nn.functional as Ffrom transformers import GPT2Config, GPT2LMHeadModel def distillation_loss(student_logits, teacher_logits, temperature=2.0):    if temperature <= 0:        raise ValueError("temperature must be positive")    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)    return (        F.kl_div(student_log_probs, teacher_probs, reduction="none")        .sum(dim=-1)        .mean()        * temperature**2    ) torch.manual_seed(0)vocab_size = 128teacher = GPT2LMHeadModel(GPT2Config(    vocab_size=vocab_size, n_layer=4, n_head=4, n_embd=64,)).eval()student = GPT2LMHeadModel(GPT2Config(    vocab_size=vocab_size, n_layer=2, n_head=4, n_embd=64,)).train()optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)tokens = torch.randint(0, vocab_size, (4, 24)) with torch.no_grad():    teacher_logits = teacher(tokens).logits[:, :-1, :]student_logits = student(tokens).logits[:, :-1, :]hard_loss = F.cross_entropy(    student_logits.reshape(-1, vocab_size), tokens[:, 1:].reshape(-1))soft_loss = distillation_loss(student_logits, teacher_logits)loss = 0.5 * hard_loss + 0.5 * soft_lossoptimizer.zero_grad()loss.backward()optimizer.step()print("training loss:", loss.item())

For a real comparison, train on a representative data split and report held-out task quality, parameter bytes, peak memory, prefill latency, and decode throughput for the same prompts and hardware. Pruning or distillation can change behavior even when parameter count falls substantially.

Further Reading #

Below are some resources you may find useful:

Summary #

In this post, you learned that compression can change model structure, not only number formats. Unstructured pruning needs sparse storage and kernels to improve runtime, while structured pruning can produce smaller dense shapes. Distillation transfers behavior into a smaller student model.

── more in #machine-learning 4 stories · sorted by recency
── more on @pytorch 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/model-compression-be…] indexed:0 read:9min 2026-11-19 · —