A new research paper argues that layer dropout, also known as stochastic depth, should not be abandoned in large language model training, citing benefits such as faster training, higher accuracy, and robustness to zero-shot layer pruning in transformers. The authors note that layer dropout has largely disappeared from large-scale models as datasets have grown, and they propose optimizing layer sparsity to improve efficiency in both training and inference.
Layer dropout (a.k.a. stochastic depth) has been shown to enable faster training, higher accuracy, and robustness to zero-shot layer pruning in both language and vision transformers. However, as models and datasets have scaled, dropout - particularly layer dropout - has largely disappeared from larg
EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.