{"slug": "model-compression-beyond-quantization", "title": "Model Compression Beyond Quantization", "summary": "A tutorial on model compression beyond quantization details how pruning, structured pruning and knowledge distillation can shrink models for cheaper inference, illustrating magnitude pruning with a PyTorch example that masks a 1024x1024 weight matrix at 50% sparsity and converts it to compressed sparse row (CSR) format. The post notes that sparse representations save space only when metadata costs less than the removed data, and speed up execution only when the matrix shape, sparsity level, dtype and hardware have an efficient sparse kernel. Structured pruning removes complete units such as attention heads and MLP channels, while distillation trains a smaller student model to reproduce a teacher's behavior.", "body_md": "Quantization changes how model values are represented. Other compression methods reduce the number of values or the size of model structures. Pruning removes weights or structures, while knowledge distillation trains a smaller student to reproduce a teacher’s behavior.\n\nServing a smaller model is often an effective inference optimization. It reads fewer weights and may use less cache, depending on its layer count, KV heads, and head dimension. It may also allow larger batches or cheaper hardware. The difficult part is preserving the behavior the application needs and ensuring that the compressed structure is supported efficiently by the runtime.\n\nIn this post, you will learn about:\n\n- How pruning removes weights and when sparsity can improve execution\n- How structured pruning changes model shapes and compute\n- How distillation trains a smaller model for serving\n\nLet’s get started.\n\n## Overview\n\nThis post is divided into three parts; they are:\n\n- Pruning and Sparsity\n- Structured Pruning\n- Knowledge Distillation\n\n## Pruning and Sparsity\n\nPruning sets selected model parameters to zero or removes larger structures. Magnitude pruning starts with a simple assumption: weights with small absolute values may have less effect on the output.\n\n``` python\nApply magnitude pruning to a linear layer and compare its output.\n\nimport torch\n\ntorch.manual_seed(0)\n\ndef magnitude_mask(weight, sparsity):\n    assert 0.0 <= sparsity < 1.0\n    if sparsity == 0.0:\n        return torch.ones_like(weight, dtype=torch.bool)\n    threshold = torch.quantile(weight.abs().float(), sparsity)\n    return weight.abs() > threshold\n\nweight = torch.randn(1024, 1024)\nmask = magnitude_mask(weight, sparsity=0.5)\npruned_weight = weight * mask\nprint(\"remaining fraction:\", mask.float().mean().item())\n\n# Convert the pruned matrix to compressed sparse row (CSR) format.\nsparse_weight = pruned_weight.to_sparse_csr()\n# Multiply a dense batch by the sparse weight matrix.\nx = torch.randn(8, 1024)\nsparse_output = torch.sparse.mm(sparse_weight, x.T).T\ndense_output = x @ pruned_weight.T\ntorch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4)\n\n1234567891011121314151617181920212223\n\nimport torch torch.manual_seed(0) def magnitude_mask(weight, sparsity):    assert 0.0 <= sparsity < 1.0    if sparsity == 0.0:        return torch.ones_like(weight, dtype=torch.bool)    threshold = torch.quantile(weight.abs().float(), sparsity)    return weight.abs() > threshold weight = torch.randn(1024, 1024)mask = magnitude_mask(weight, sparsity=0.5)pruned_weight = weight * maskprint(\"remaining fraction:\", mask.float().mean().item()) # Convert the pruned matrix to compressed sparse row (CSR) format.sparse_weight = pruned_weight.to_sparse_csr()# Multiply a dense batch by the sparse weight matrix.x = torch.randn(8, 1024)sparse_output = torch.sparse.mm(sparse_weight, x.T).Tdense_output = x @ pruned_weight.Ttorch.testing.assert_close(sparse_output, dense_output, rtol=1e-4, atol=2e-4)\n```\n\nSparse and dense matrix multiplication can accumulate terms in a different order, so the script compares their outputs with a numerical tolerance. The `sparsity` argument represents the fraction of weights intended for removal. Setting `sparsity=0.0` retains all weights. The upper bound excludes `sparsity=1.0`, which would remove every weight and make the layer unusable. After pruning, zeros can appear anywhere in `pruned_weight`, but that tensor is still stored densely. Calling `to_sparse_csr()` creates a compressed sparse row tensor that stores the nonzero values together with row and column metadata. The function `torch.sparse.mm()` then multiplies that sparse matrix by a dense matrix. A sparse representation saves space only when its metadata costs less than the removed data, and it improves speed only when the matrix shape, sparsity level, dtype, and hardware have an efficient sparse kernel.\n\n## Structured Pruning\n\nStructured pruning removes complete units such as attention heads, MLP channels, layers, or blocks. The result can often be represented as a smaller dense model, which makes it easier to accelerate across hardware.\n\nA transformer MLP uses one projection to expand each hidden vector into an intermediate representation and another to project it back to the model’s hidden size:\n\n$$\n\n\\mathbf{y} = \\mathbf{W}_{\\text{down}} \\phi(\\mathbf{W}_{\\text{up}} \\mathbf{x})\n\n$$\n\nEach intermediate channel is produced by one row of $\\mathbf{W}_{\\text{up}}$ and consumed by the matching column of $\\mathbf{W}_{\\text{down}}$. Removing a row from the up projection therefore requires removing the corresponding column from the down projection. The following small example removes intermediate channels according to the norms of the first projection’s rows:\n\n``` python\nRemove low-norm intermediate channels from a two-layer projection.\n\nimport torch\nimport torch.nn as nn\n\ndef shrink_mlp(up_proj, down_proj, keep_channels):\n    if not 1 <= keep_channels <= up_proj.out_features:\n        raise ValueError(\"keep_channels must be within the MLP width\")\n    if up_proj.out_features != down_proj.in_features:\n        raise ValueError(\"projection widths must match\")\n    scores = up_proj.weight.norm(dim=1)\n    keep = scores.topk(keep_channels).indices.sort().values\n\n    smaller_up = nn.Linear(\n        up_proj.in_features,\n        keep_channels,\n        bias=up_proj.bias is not None,\n    ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype)\n    smaller_down = nn.Linear(\n        keep_channels,\n        down_proj.out_features,\n        bias=down_proj.bias is not None,\n    ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype)\n\n    with torch.no_grad():\n        smaller_up.weight.copy_(up_proj.weight[keep])\n        smaller_down.weight.copy_(down_proj.weight[:, keep])\n        if up_proj.bias is not None:\n            smaller_up.bias.copy_(up_proj.bias[keep])\n        if down_proj.bias is not None:\n            smaller_down.bias.copy_(down_proj.bias)\n\n    return smaller_up, smaller_down\n\nup = nn.Linear(16, 32)\ndown = nn.Linear(32, 16)\nsmaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)\nprint(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape)\n\n123456789101112131415161718192021222324252627282930313233343536\n\nimport torchimport torch.nn as nn def shrink_mlp(up_proj, down_proj, keep_channels):    if not 1 <= keep_channels <= up_proj.out_features:        raise ValueError(\"keep_channels must be within the MLP width\")    if up_proj.out_features != down_proj.in_features:        raise ValueError(\"projection widths must match\")    scores = up_proj.weight.norm(dim=1)    keep = scores.topk(keep_channels).indices.sort().values     smaller_up = nn.Linear(        up_proj.in_features,        keep_channels,        bias=up_proj.bias is not None,    ).to(device=up_proj.weight.device, dtype=up_proj.weight.dtype)    smaller_down = nn.Linear(        keep_channels,        down_proj.out_features,        bias=down_proj.bias is not None,    ).to(device=down_proj.weight.device, dtype=down_proj.weight.dtype)     with torch.no_grad():        smaller_up.weight.copy_(up_proj.weight[keep])        smaller_down.weight.copy_(down_proj.weight[:, keep])        if up_proj.bias is not None:            smaller_up.bias.copy_(up_proj.bias[keep])        if down_proj.bias is not None:            smaller_down.bias.copy_(down_proj.bias)     return smaller_up, smaller_down up = nn.Linear(16, 32)down = nn.Linear(32, 16)smaller_up, smaller_down = shrink_mlp(up, down, keep_channels=16)print(smaller_down(torch.relu(smaller_up(torch.randn(2, 16)))).shape)\n```\n\nThe parameter `keep_channels` is the number of intermediate MLP channels retained after pruning; it is a count, not a sparsity fraction. It must be an integer from 1 through `up_proj.out_features`, and the original projections must satisfy `up_proj.out_features == down_proj.in_features`. The code ranks the rows of `up_proj.weight` by their norms and keeps the `keep_channels` highest-scoring rows. It then keeps the matching columns of `down_proj.weight`, so the reduced output of `smaller_up` remains a valid input to `smaller_down`. For example, setting `keep_channels=2048` reduces an original intermediate width of 8192 to 2048 channels.\n\nThis code preserves compatible tensor shapes but does not determine whether the chosen channels are truly unimportant. A real pruning method evaluates layer outputs, interactions, and recovery fine-tuning.\n\nRemoving attention heads requires similar coordinated edits to query, key, value, and output projections. With GQA, query heads share KV heads, so arbitrary head removal can break grouping. Removing transformer layers changes residual depth and may require learning mappings between the retained layers. These changes require coordinated edits and quality checks.\n\nStructured pruning can reduce parameter count and dense compute, but awkward hidden sizes may perform worse than slightly larger hardware-friendly sizes. Align channel counts with the tile requirements of the target matrix multiplication kernels.\n\n## Knowledge Distillation\n\nA smaller model can run faster and use less memory. Knowledge distillation trains a smaller student model using signals from a larger teacher. The student may learn from the teacher’s output distribution (not just the highest-scoring token), intermediate representations, generated data, or a combination of them.\n\nFor logits $\\mathbf{z}_t$ and $\\mathbf{z}_s$ from teacher and student, a common distillation loss function uses a temperature $T$:\n\n$$\n\n\\mathcal{L}_{KD} = T^2\\operatorname{KL}\\Big(\n\n\\operatorname{softmax}\\big(\\frac{\\mathbf{z}_t}{T}\\big)\\;\\big\\Vert \\;\\operatorname{softmax}\\big(\\frac{\\mathbf{z}_s}{T}\\big)\n\n\\Big)\n\n$$\n\nHere, $\\operatorname{KL}(\\mathbf{P}\\Vert \\mathbf{Q})$ is the Kullback-Leibler divergence from the teacher distribution $\\mathbf{P}$ to the student distribution $\\mathbf{Q}$; it is zero when they match and grows as their probabilities differ. The corresponding PyTorch code is:\n\n```\nCompute a temperature-scaled distillation loss from teacher and student logits.\n\nimport torch\nimport torch.nn.functional as F\n\ndef distillation_loss(student_logits, teacher_logits, temperature=2.0):\n    if temperature <= 0:\n        raise ValueError(\"temperature must be positive\")\n    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)\n    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)\n    return (\n        F.kl_div(student_log_probs, teacher_probs, reduction=\"none\")\n        .sum(dim=-1)\n        .mean()\n        * temperature**2\n    )\n\nstudent_logits = torch.tensor([[1.0, 0.0]])\nteacher_logits = torch.tensor([[0.0, 1.0]])\nprint(\"distillation loss:\", distillation_loss(student_logits, teacher_logits).item())\n\n123456789101112131415161718\n\nimport torchimport torch.nn.functional as F def distillation_loss(student_logits, teacher_logits, temperature=2.0):    if temperature <= 0:        raise ValueError(\"temperature must be positive\")    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)    return (        F.kl_div(student_log_probs, teacher_probs, reduction=\"none\")        .sum(dim=-1)        .mean()        * temperature**2    ) student_logits = torch.tensor([[1.0, 0.0]])teacher_logits = torch.tensor([[0.0, 1.0]])print(\"distillation loss:\", distillation_loss(student_logits, teacher_logits).item())\n```\n\nFor padded batches, replace `.mean()` with a masked average over valid token positions. This loss assumes teacher and student logits refer to the same vocabulary and token positions. The teacher runs without gradients, but producing its targets still costs time. The next example defines the same loss so it can run independently.\n\nThe following local example uses two small, randomly initialized GPT-2 style models. It demonstrates one training step; it does *not* produce a useful language model. A real distillation experiment needs a pretrained teacher and representative text.\n\n```\nRun one student training step with a temperature-scaled distillation loss.\n\nimport torch\nimport torch.nn.functional as F\nfrom transformers import GPT2Config, GPT2LMHeadModel\n\ndef distillation_loss(student_logits, teacher_logits, temperature=2.0):\n    if temperature <= 0:\n        raise ValueError(\"temperature must be positive\")\n    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)\n    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)\n    return (\n        F.kl_div(student_log_probs, teacher_probs, reduction=\"none\")\n        .sum(dim=-1)\n        .mean()\n        * temperature**2\n    )\n\ntorch.manual_seed(0)\nvocab_size = 128\nteacher = GPT2LMHeadModel(GPT2Config(\n    vocab_size=vocab_size, n_layer=4, n_head=4, n_embd=64,\n)).eval()\nstudent = GPT2LMHeadModel(GPT2Config(\n    vocab_size=vocab_size, n_layer=2, n_head=4, n_embd=64,\n)).train()\noptimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)\ntokens = torch.randint(0, vocab_size, (4, 24))\n\nwith torch.no_grad():\n    teacher_logits = teacher(tokens).logits[:, :-1, :]\nstudent_logits = student(tokens).logits[:, :-1, :]\nhard_loss = F.cross_entropy(\n    student_logits.reshape(-1, vocab_size), tokens[:, 1:].reshape(-1)\n)\nsoft_loss = distillation_loss(student_logits, teacher_logits)\nloss = 0.5 * hard_loss + 0.5 * soft_loss\noptimizer.zero_grad()\nloss.backward()\noptimizer.step()\nprint(\"training loss:\", loss.item())\n\n123456789101112131415161718192021222324252627282930313233343536373839\n\nimport torchimport torch.nn.functional as Ffrom transformers import GPT2Config, GPT2LMHeadModel def distillation_loss(student_logits, teacher_logits, temperature=2.0):    if temperature <= 0:        raise ValueError(\"temperature must be positive\")    teacher_probs = F.softmax(teacher_logits / temperature, dim=-1)    student_log_probs = F.log_softmax(student_logits / temperature, dim=-1)    return (        F.kl_div(student_log_probs, teacher_probs, reduction=\"none\")        .sum(dim=-1)        .mean()        * temperature**2    ) torch.manual_seed(0)vocab_size = 128teacher = GPT2LMHeadModel(GPT2Config(    vocab_size=vocab_size, n_layer=4, n_head=4, n_embd=64,)).eval()student = GPT2LMHeadModel(GPT2Config(    vocab_size=vocab_size, n_layer=2, n_head=4, n_embd=64,)).train()optimizer = torch.optim.AdamW(student.parameters(), lr=1e-3)tokens = torch.randint(0, vocab_size, (4, 24)) with torch.no_grad():    teacher_logits = teacher(tokens).logits[:, :-1, :]student_logits = student(tokens).logits[:, :-1, :]hard_loss = F.cross_entropy(    student_logits.reshape(-1, vocab_size), tokens[:, 1:].reshape(-1))soft_loss = distillation_loss(student_logits, teacher_logits)loss = 0.5 * hard_loss + 0.5 * soft_lossoptimizer.zero_grad()loss.backward()optimizer.step()print(\"training loss:\", loss.item())\n```\n\nFor a real comparison, train on a representative data split and report held-out task quality, parameter bytes, peak memory, prefill latency, and decode throughput for the same prompts and hardware. Pruning or distillation can change behavior even when parameter count falls substantially.\n\n## Further Reading\n\nBelow are some resources you may find useful:\n\n- Jonathan Frankle, Michael Carbin (2018). [The Lottery Ticket Hypothesis](https://arxiv.org/abs/1803.03635) . This paper studies sparse subnetworks that can train effectively within larger networks.\n- Elias Frantar, Dan Alistarh (2023). [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://arxiv.org/abs/2301.00774) . This paper presents a post-training pruning method for large language models.\n- Geoffrey Hinton, Oriol Vinyals, Jeff Dean (2015). [Distilling the Knowledge in a Neural Network](https://arxiv.org/abs/1503.02531) . This paper introduces the commonly used soft-target distillation formulation.\n- Victor Sanh et al. (2019). [DistilBERT](https://arxiv.org/abs/1910.01108) . This paper applies knowledge distillation while pretraining a smaller transformer.\n\n## Summary\n\nIn this post, you learned that compression can change model structure, not only number formats. Unstructured pruning needs sparse storage and kernels to improve runtime, while structured pruning can produce smaller dense shapes. Distillation transfers behavior into a smaller student model.", "url": "https://wpnews.pro/news/model-compression-beyond-quantization", "canonical_source": "https://machinelearningmastery.com/model-compression-beyond-quantization/", "published_at": "2026-11-19 14:06:50+00:00", "updated_at": "2026-11-19 20:12:54.852577+00:00", "lang": "en", "topics": ["machine-learning", "ai-infrastructure", "large-language-models", "ai-tools"], "entities": ["PyTorch", "torch.sparse.mm", "to_sparse_csr"], "also_reported_by": [], "takeaway": "Magnitude pruning at 50% sparsity on a 1024x1024 linear layer matches dense output within rtol=1e-4 and atol=2e-4 via torch.sparse.mm, but only pays off when the CSR metadata costs less than the removed weights and the hardware has an efficient sparse kernel.", "recommendation": {"advice": "Convert pruned weight matrices to CSR with to_sparse_csr() and verify sparse versus dense outputs within tolerance, but only rely on sparsity for speed when the shape, dtype and hardware have an efficient sparse kernel.", "playbook": "inference-serving", "playbook_url": "https://wpnews.pro/playbooks/inference-serving"}, "alternates": {"html": "https://wpnews.pro/news/model-compression-beyond-quantization", "markdown": "https://wpnews.pro/news/model-compression-beyond-quantization.md", "text": "https://wpnews.pro/news/model-compression-beyond-quantization.txt", "jsonld": "https://wpnews.pro/news/model-compression-beyond-quantization.jsonld"}}