Backprop Has a Real Challenger: Dust Trains Transformers
Researchers at QLabs published Dust, a zeroth-order optimization method that matches backpropagation on transformer pretraining at scale, reaching a test loss of approximately 5.17 at 1 million tokens…