SCOPD: Sparse-Context On-Policy Self-Distillation for Efficient Vision-Language Models

Researchers introduced SCOPD (Sparse-Context On-Policy Self-Distillation), a method for efficient vision-language models that addresses the performance degradation caused by training-free token pruning of long visual-token sequences. The approach targets the irreversible loss of task-relevant visual information that occurs under aggressive compression, which makes inference expensive for reasoning VLMs processing images and videos.

Reasoning vision-language models VLMs process images and videos as long sequences of visual tokens, making inference expensive. Training-free token pruning reduces this cost, but aggressive compression can sharply degrade performance, often attributed to irreversible loss of task-relevant visual i