Importance-Aware Low-Rank Distillation of Diffusion Transformers

Researchers introduced SVDtrunc, a two-step block-level compression scheme for Diffusion Transformers (DiTs), which outperforms competing approaches across GenEval, HPSv2, and DPG benchmarks while retaining near-full performance at 68% and remaining competitive at 57% of the original parameter budget. The method, applied to FLUX.dev, combines truncated singular value decomposition with modular knowledge distillation and a rectified-flow objective, and complements step distillation for efficient text-to-image generation.

arXiv:2609.04646v1 Announce Type: new Abstract: Diffusion Transformers DiTs have emerged as a dominant architecture for high-quality text-to-image generation, yet their scale poses challenges for efficient deployment. While truncated singular value decomposition SVD is a principled tool for parameter reduction, evidence from large language models LLMs suggests that naive low-rank approximation can cause catastrophic failure. In contrast, we find that truncated SVD in DiTs produces smooth degradation even under substantial global compression, with redundancy distributed across projection matrices throughout the whole network rather than concentrated in a few transformer blocks. Building on these insights, we introduce SVDtrunc, a two-step block-level compression scheme, first allocating ranks across blocks and compressing the least important ones via truncated SVD under a global parameter budget, and then fine-tuning all blocks with modular knowledge distillation and a rectified-flow objective. We apply SVDtrunc to FLUX.dev across compression levels ranging from 40-90% of the original parameter count. Across three benchmarks, GenEval, HPSv2, and DPG, we outperform all competing approaches. Notably, and in contrast to prior work, we retain near-full performance at 68% and remain competitive even at 57% of the original parameter budget. Furthermore, we show that SVDtrunc complements step distillation and achieves strong results even without fine-tuning, positioning it as a practical continuation of efficiency improvements beyond diffusion step reduction for large-scale generative models. Project page: https://vislearn.github.io/SVDtrunc/