Don't Drop Dropout: Optimizing Layer Sparsity for Efficient LLM Training and Inference

A new research paper argues that layer dropout, also known as stochastic depth, should not be abandoned in large language model training, citing benefits such as faster training, higher accuracy, and robustness to zero-shot layer pruning in transformers. The authors note that layer dropout has largely disappeared from large-scale models as datasets have grown, and they propose optimizing layer sparsity to improve efficiency in both training and inference.

Layer dropout a.k.a. stochastic depth has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from larg