Noise is a pretraining speedrun for learning algorithms that never run a backward pass. Model, data and score are fixed; two tracks ask how low the loss can go and how fast. Our Dust baseline sits at 4.167.
Backprop is the only credit assignment algorithm that trains modern neural nets, and the field has co-evolved around it: it needs differentiability and gives first-order gradients, and our architectures, optimizers and hardware are built around it. As compute keeps growing, brute-force learning algorithms based on search may overtake it. The bitter lesson [1]Richard S. Sutton. The Bitter Lesson, 2019. URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html. is that general methods which scale with compute win; AlphaGo Zero [2]David Silver et al. Mastering the game of Go without human knowledge. Nature, 550:354–359, 2017. URL https://doi.org/10.1038/nature24270. is the obvious example: bootstrapping on human games helped at first, and pure self-play won once there was enough compute. Differentiability and backprop may be the same kind of bias: efficient while compute is scarce, limiting once there is enough of it. They constrain which architectures work, and even for the ones that do, they explore the loss landscape poorly, which may be why today's networks need so much data to generalize.
We think replacing backprop with a much better credit assignment algorithm is inevitable. Noise tests that on the hardest task we know, pretraining language models, and scores two things: how low the validation loss goes given a lot of compute, and how fast it gets there. We just published Dust [3]Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna. Dust: Pretraining Transformers Without Backpropagation, 2026. URL https://qlabs.sh/research/dust/., our attempt to replace backprop with brute-force computation rather than analytic structure. Dust perturbs activations [4]Justin Werfel, Xiaohui Xie and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. NeurIPS, 2003. instead of weights [5]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor and Ilya Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv preprint arXiv:1703.03864, 2017. URL https://arxiv.org/abs/1703.03864., independently at every token, so one forward pass evaluates thousands of population members, and the reward-weighted noise becomes a gradient estimate for an ordinary optimizer. Activations are where interpretability work finds the model's reasoning [6, 7]Jack Lindsey et al. On the Biology of a Large Language Model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html.
Wes Gurnee et al. Verbalizable representations form a global workspace in language models. arXiv preprint arXiv:2607.15495, 2026. URL https://arxiv.org/abs/2607.15495., so training becomes a search over latent reasoning [8]Akshay Vegesna and Samip Dahal. Decoupling Search and Learning in Neural Net Training. arXiv preprint arXiv:2509.10973, 2025. URL https://arxiv.org/abs/2509.10973..
Dust is 1,000 to 10,000x more efficient than EGGROLL [9]Bidipta Sarkar et al. Evolution Strategies at the Hyperscale. arXiv preprint arXiv:2511.16652, 2025. URL https://arxiv.org/abs/2511.16652., the state-of-the-art weight-space ES, and the first method to rival backprop with SGD, matching it at large populations and beating it in several settings, when the consensus was that this is not possible. With AdamW it is still behind, and we think another 1,000x on top of Dust is possible. That will take co-designing the optimizer, the schedule and the credit assignment, and probably the architecture and hardware too. We keep the last two fixed for now, for simplicity, and will relax that soon.
Setting #
Everything but the learning algorithm is pinned to our paper's 10M-token setting: its 37.7M-parameter GPT, which the final weights must load into exactly; its 10M-token FineWeb [10]Guilherme Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS Datasets and Benchmarks, 2024. URL https://arxiv.org/abs/2406.17557. training set, for at most one epoch; and the cross-entropy of the final weights on its validation set. The only other rule is no backpropagation: no autograd in either mode and no hand-written backward pass. Initialization, batch size and how you use your forward passes are free. Records are PRs; the full rules are in the repo [11]Q Labs. Noise, 2026. URL https://github.com/qlabs-eng/noise..
| Track | Objective | Current record |
|---|---|---|
| 1. Validation loss | Lowest validation loss after one epoch, no time limit | 4.167 in 1.76 h, Dust with AdamW |
| 2. Speedrun | Shortest training time to validation loss 4.2 | 1.76 h, Dust with AdamW |
Update: 4,000x Faster #
In its first ten days, the community made Noise's speedrun over 4,000x faster: the time to reach validation loss 4.2 without a backward pass fell from 1.76 h to 1.6 s. We wrote that another 1,000x on top of Dust was possible. We didn't expect the speedrun to pass that in ten days, and we're grateful to everyone who submitted.
Track 1 moved fast too, from 4.167 to 3.676, past backprop with Adam (3.769). Backprop with the same optimizer is still ahead, for now.
Some highlights:
- @Lazarus-931 brought in the ANVIL II optimizer and, later, identity-path credit, which measures credit once at the top of the network and hands it down to every block.
- @JaydenTeoh made Dust's estimates cleaner with orthogonal draws and an associative credit memory.
- @spicylemonade replaced Dust's random draws with a full Hadamard cycle in Boost → Cut, measuring every neuron's credit at every token without a population, the jump behind both tracks.
- @hoo0681 ,@CharlesDDDD and@NotDesigned won back credit that Cut loses by freezing attention, with own queries, a value path, future-token probes and first-hop credit. Future-token probes took track 1 past backprop with Adam.
- @CharlesDDDD then set the current track 1 record, adding an end merge of weight averages.
- @pculaf ,@jon123boss and@isaac0804 sped the speedrun up step by step with better probes and kernels.
- @RobertTLange sharded the work across GPUs, wrote the moment kernels the latest records build on, and later made each update much cheaper.
- @erodola and@Euraba took it further with a structured start, tail averaging, 8-row updates, overlapped communication and faster kernels.
- @areu01or00 added a smarter finish and then doubled the rows per update, setting the current record.
While this is very exciting, scaling will require more fundamental changes. Speedruns are great at iterating, and most records so far refine the one before, tuned to this one setting. We believe scalable gains will come from more creative approaches, one of them being optimizer-estimator codesign. To reward those, we'll keep raising the bar as the tracks saturate, starting later today: we are moving both from 10M to 30M training tokens, with more exciting changes coming soon.
If you're working on any of this, open a PR or an issue on the [repo](https://github.com/qlabs-eng/noise), or email [research@qlabs.sh](mailto:research@qlabs.sh).
References #
[1]Richard S. Sutton. The Bitter Lesson, 2019. URL http://www.incompleteideas.net/IncIdeas/BitterLesson.html.
[2]David Silver et al. Mastering the game of Go without human knowledge. Nature, 550:354–359, 2017. URL https://doi.org/10.1038/nature24270.
[3]Samip Dahal, Bishwas Mandal, Serdar Gülbahar and Akshay Vegesna. Dust: Pretraining Transformers Without Backpropagation, 2026. URL https://qlabs.sh/research/dust/.
[4]Justin Werfel, Xiaohui Xie and H. Sebastian Seung. Learning curves for stochastic gradient descent in linear feedforward networks. NeurIPS, 2003.
[5]Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor and Ilya Sutskever. Evolution Strategies as a Scalable Alternative to Reinforcement Learning. arXiv preprint arXiv:1703.03864, 2017. URL https://arxiv.org/abs/1703.03864.
[6]Jack Lindsey et al. On the Biology of a Large Language Model. Transformer Circuits Thread, 2025. URL https://transformer-circuits.pub/2025/attribution-graphs/biology.html.
[7]Wes Gurnee et al. Verbalizable representations form a global workspace in language models. arXiv preprint arXiv:2607.15495, 2026. URL https://arxiv.org/abs/2607.15495.
[8]Akshay Vegesna and Samip Dahal. Decoupling Search and Learning in Neural Net Training. arXiv preprint arXiv:2509.10973, 2025. URL https://arxiv.org/abs/2509.10973.
[9]Bidipta Sarkar et al. Evolution Strategies at the Hyperscale. arXiv preprint arXiv:2511.16652, 2025. URL https://arxiv.org/abs/2511.16652. [10]Guilherme Penedo et al. The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale. NeurIPS Datasets and Benchmarks, 2024. URL https://arxiv.org/abs/2406.17557.
[11]Q Labs. Noise, 2026. URL https://github.com/qlabs-eng/noise.