GuppyLM is a small language model that talks like a fish. The repository is small enough for a beginner to read from end to end. It shows the path from data and tokenization to training and inference without hiding the pieces inside a large framework.
It is slightly more complex than Karpathy’s microgpt. That is useful. GuppyLM is not an implement-everything-from-scratch demo. It uses PyTorch, separates the model, dataset, training loop, and inference code, and looks closer to the code used to train and serve models today.
The model is still simple: 8.7 million parameters, six Transformer layers, six attention heads, a 4,096-token vocabulary, and a 128-token context window. The training code includes AdamW, learning-rate warmup and cosine decay, mixed precision, gradient clipping, evaluation, and checkpoints. The inference code loads the tokenizer and checkpoint, then generates tokens with temperature and top-k sampling.
The model trains on 60,000 synthetic conversations. The project says training takes about five minutes on one GPU. Its quantized ONNX export is about 10 MB and runs in a browser. This makes the whole loop fast enough to inspect, change, train, and test instead of only reading about it.