In a technical explainer, the author demonstrates that L2 regularization and weight decay, equivalent in plain SGD, diverge under the Adam optimizer because Adam's adaptive step rescales the L2 penalt…
A guide explains that a neuron is a weighted sum followed by a non-linear activation, without which a deep network would be a linear regression. The training workflow consists of forward pass, loss ca…
A technical blog post explains that AdamW with warmup and cosine decay is the optimizer almost everyone uses today to train large language models, tracing the evolution from vanilla SGD through moment…
A new study introduces the Guided-Retry strategy to reduce AI hallucinations in task-oriented dialogues without retraining models. Tested on models like DeepSeek-R1 and Llama-3, the method cut halluci…
Researchers introduce Depth-wise Gradient Augmentation, a new optimization paradigm that transforms layer-wise updates to exploit structured relationships across deep neural network layers. Their inst…
Researchers closed a gap in high-probability bounds for stochastic gradient descent under the Polyak-Łojasiewicz condition with Markovian noise, proving optimal linear dependence on mixing time. They …