RMSNorm, a normalization technique proposed by Zhang and Sennrich in 2019, rescales neural network activations by dividing by the root mean square of each row, skipping the mean-centering step of Laye…
A 3.16M-parameter transformer running on a $250 FPGA achieves 44 tokens per second by moving the CPU out of the per-token loop, according to a technical blog post detailing the on-chip LLM design. The…
A developer implemented batch normalization, layer normalization, and group normalization from scratch and compared their performance on a simple multi-layer perceptron (MLP) classifying the MNIST dat…
A user investigating GPT-2's last hidden state in Hugging Face Transformers 3.1.0 found that the final hidden state (layer 12) is reported after the last decoder block but before the final LayerNorm (…
A developer trained a 30M-parameter decoder-only transformer from scratch on the TinyStories dataset using Kaggle's free T4 GPUs, achieving a validation loss of 1.401 at a learning rate of 1e-3. The p…
A developer explains the difference between LayerNorm and BatchNorm, showing that LayerNorm normalizes per token rather than per batch, which makes it ideal for Transformers. The post includes an inte…
A new study proves the Spectral Alignment Decomposition, which explains why the curvature exponent $\alpha$ — governing how Hessian eigenvalues scale with gradient singular values — varies across neur…