Researchers improved associative recall in recurrent neural networks by orthogonalizing the mLSTM memory matrix during reads, inspired by the Muon optimizer. In noisy associative recall tasks, the met…
Researchers introduce Depth-wise Gradient Augmentation, a new optimization paradigm that transforms layer-wise updates to exploit structured relationships across deep neural network layers. Their inst…
Researchers introduced Muon$^p$, a new optimizer that uses fractional spectral-power updates to interpolate between Muon and gradient descent, improving finetuning performance on billion-scale models.…
Vlad Feinberg published a guide on how to land a job at a frontier AI lab focused on pretraining, emphasizing kernel-level performance work as the most direct path into the labs. The guide recommends …
DeepSeek-V4's million-token context capability stems from a hybrid attention architecture that compresses context before KV storage, reducing cache pressure. Together's early bring-up on NVIDIA HGX B2…
DeepSeek-V4 shifts the focus from simply offering a million-token context window to making that window practically usable through efficient attention mechanisms. Its architecture employs hybrid compre…