Erasing a Bit Warms the Room
A developer built a tiny language model with 50 words of vocabulary and 1,600 parameters to show that the loss number reported during training is a physical quantity measured in bits, not just a score. The model's perple…