Most model families ship in a few fixed sizes: small, base, large. If you pick one for typical traffic, it is too slow at peak and wasteful when load is light. "Your Transformer Is Secretly Elastic" (Simoncini, Varghese, Burghouts and Snoek, University of Amsterdam and TNO, posted 14 October 2026) proposes turning a single pretrained transformer encoder into a model that can run at many compute budgets, chosen at inference time, with no retraining and no weight updates.
The approach is structured pruning with a better ranking. Every MLP neuron and every attention head gets an importance score. To run at a given budget, you keep the top-ranked units until their cumulative FLOP cost reaches the target fraction of the dense model, and drop the rest. The model weights never change. The entire contribution is in how the ranking is produced.
Scores start from a second-order view of importance: the loss increase from removing a parameter is approximated by the diagonal of the Hessian, estimated as the empirical Fisher, meaning the mean of squared gradients over a small calibration set. Per-weight scores are averaged into per-unit scores, such as the input column and output row of an MLP neuron. For vision encoders the calibration loss is the self-supervised DINO objective, which works for supervised and self-supervised backbones alike. For text encoders it is a symmetrized in-batch InfoNCE loss. This follows earlier pruning work such as Kwon et al. (2022).
The authors list three weaknesses of these scores. The diagonal approximation ignores interactions between units, the proxy loss may not match the downstream task, and scores are not comparable across layers or between neurons and heads.
The fix is to optimize the scores directly. Hard pruning is a threshold on a ranking and is not differentiable, so the paper relaxes it. Scores are stored as learnable log-values, centered by their median per unit type so neurons and heads share a common scale. For a sparsity target b, the method finds the threshold at which the kept units reach the FLOP budget, then replaces the hard keep-or-drop decision with a sigmoid of the distance from that threshold, scaled by a temperature. The soft mask multiplies MLP activations and attention head outputs in the forward pass.
The objective averages the masked model's loss across 14 sparsity targets, from 5% to 70% in 5% steps, using the same calibration batch. Only the log-scores are updated by gradient descent. Because mask gradients are largest near each threshold, different targets expose different parts of the ranking to useful gradients. After optimization, the ranking can be hard-pruned to any budget without further work.
An earlier method from the same group optimized one scaling factor per block and head using evolutionary search. That kept the neuron ordering inside each MLP block fixed, so keeping one important but low-scored neuron meant keeping every neuron ranked above it. Per-neuron scores remove that constraint.
The evaluation covers 27 vision and text transformer encoders from 22M to 8B parameters, across image classification, semantic segmentation, text embedding and retrieval. Numbers below are from the paper:
The practical appeal is operational. Serving k budgets with ordinary pruning means producing, storing and switching among k separate models. For retrieval there is an additional cost, since embeddings from different pruned models may not be compatible, so you may need separate indices. A single ranking over one set of weights avoids both, and the authors note compatibility with a fixed dense index. A serving layer could then use the full model when load is low and a pruned configuration during a spike.
It also contrasts with other routes to elasticity. Nested designs such as MatFormer need a particular architecture and training from scratch, and other methods need substantial fine-tuning. Both are expensive when you only have a released checkpoint, or when you do not know its pretraining data. Skipping weight updates sidesteps that.
The results are for vision encoders and bidirectional text encoders. The authors state that deep pruning remains difficult for decoder-based text embedders, and they leave autoregressive language modeling and diffusion image generation to future work. Anyone hoping to apply this to a chat model should treat that as unproven.
A 1.80x throughput gain at 60% sparsity is real but well short of the FLOP reduction, and the single-query latency gain is only 1.26x. Structured pruning leaves smaller matrices, which GPUs do not always use efficiently, so measured gains depend on batch size and hardware. The paper includes a dedicated analysis of whether pruning translates to speedups, which is worth reading before planning capacity.
The method needs a calibration set and a loss. The paper uses DINO for vision and contrastive losses for text, and includes ablations on calibration set size and task-aligned calibration. A ranking tuned on one distribution may not hold under a very different one. Related work on importance estimation, such as Molchanov et al. (2019), has long noted that importance scores are sensitive to how they are measured.
Switching budgets at runtime means changing which units are active, which has its own kernel and memory-layout cost. The paper reports low budget-switching overhead, but a production stack with batching, caching and quantization adds interactions that a research benchmark does not test.
The core idea is simple: pretrained transformers already contain a usable family of smaller models if you rank their units well. What is still unknown is whether the same ranking trick holds for decoder language models, where the authors saw sensitivity, and whether the speedups survive a real serving stack with mixed batch sizes. For this to matter in practice, someone would need to show the elastic ranking working on the models people actually deploy at scale, with latency gains that justify the added operational complexity.
Also see the paper's Hugging Face papers listing for discussion and linked resources.