Dust: Pretraining Transformers Without Backpropagation
Researchers Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna released Dust, a zeroth-order pretraining method that perturbs activations independently at every token so one forward pass evaluates a virtual …