ProgResViT: Progressive Resolution and Width for Adaptive Vision Transformers

Researchers introduced ProgResViT, an input-adaptive Vision Transformer that progressively increases input resolution and model width across rounds, terminating early when predictions are confident. Applied to DeiT, ProgResViT achieves 84.9% top-1 accuracy with knowledge distillation, slightly exceeding DeiT-III-S, while offering better accuracy-compute trade-offs than adaptive-width, adaptive-depth, and dynamic-token baselines. The design also benefits self-supervised DINO representations and semantic segmentation, with code available on GitHub.

arXiv:2609.03216v1 Announce Type: new Abstract: Vision Transformers ViTs typically process every image using a fixed input resolution and model width, even though many images can be classified with substantially less computation. We introduce ProgResViT, an input-adaptive ViT that performs inference progressively across multiple rounds. The first round processes a low-resolution image with a narrow subnetwork. Inference terminates when the prediction is sufficiently confident; otherwise, the model reuses the representations produced in the current round and proceeds with a higher-resolution input and a wider subnetwork to refine its prediction. As all rounds share a single backbone, we propose Progress-Conditioned Soft Gating PSG , which conditions token fusion and layer outputs on the current round, block, and input resolution. On image classification, applying ProgResViT to DeiT yields better accuracy-compute trade-offs than adaptive-width, adaptive-depth, and dynamic-token baselines. With knowledge distillation, a DeiT-based ProgResViT achieves 84.9% top-1 accuracy, slightly exceeding the reported DeiT-III-S accuracy under a comparable evaluation setting. We show that the same design also provides favorable accuracy-compute trade-offs for self-supervised DINO representations and downstream semantic segmentation. Code is available at https://github.com/ds-kiel/ProgResViT.