I Trained a Language Model From Scratch on a 4 GB GPU
A developer trained a 22-million-parameter language model from scratch on a laptop with only 4 GB of VRAM, using a Lenovo IdeaPad Gaming machine with a GTX 1650, PyTorch and CUDA under WSL2. The run used a 256-token cont…