SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Researchers at arXiv have identified that Batch Normalization causes gradient skew in dynamic sparse training (DST) methods, leading to slower convergence compared to dense neural network training. The team proposed SparseOpt, a sparsity-aware optimizer that addresses this issue, demonstrating consistently faster convergence and improved generalization on ResNet models across CIFAR-100 and ImageNet. This work provides the first systematic study of the interaction between Batch Normalization and sparse layers, marking a step toward making DST practically competitive with dense training.

arXiv:2605.27541v1 Announce Type: new Abstract: Dynamic Sparse Training DST methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization BN adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.