{"slug": "building-and-training-llm-from-scratch-no-gpu", "title": "Building and Training LLM from scratch (No GPU)", "summary": "A developer built a ~10.6 million parameter decoder-only Transformer from scratch in PyTorch and trained it end-to-end on the Tiny Shakespeare dataset using only the free tier of Google Colab's T4 GPU. The 6-layer, 6-head, 384-dimensional model uses character-level tokenization with a 65-character vocabulary and trains for 5,000 steps in roughly 30–45 minutes, with loss dropping from about 4.2 to 1.5–1.8. The project, released on GitHub, produces recognizable Shakespeare-style text and is intended to demystify LLM internals by avoiding pre-trained models and APIs.", "body_md": "*How I trained a decoder-only Transformer on the Tiny Shakespeare dataset using nothing but a free Google Colab T4 GPU.*\n\nThe best way to understand how large language models work is to build one yourself. Not by fine-tuning a pre-trained model, not by calling an API, but by writing every layer, every attention head, and every line of the training loop from scratch.\n\nThis is the story of [Tiny_Shakespeare_LLM_from_scratch](https://github.com/Devn913/Tiny_Shakespeare_LLM_from_scratch) — a ~10 million parameter decoder-only Transformer built in PyTorch and trained end-to-end on the Tiny Shakespeare dataset using only the free tier of Google Colab.\n\nNo paid GPUs. No cloud credits. Just a free T4 and a few hours of training.\n\nUsing libraries like Hugging Face `transformers` is convenient, but it hides the mechanics. Building from scratch forces you to confront the questions that matter:\n\nBy the end of this project, the Transformer architecture stops being a black box. It becomes a sequence of tensor operations you wrote yourself.\n\nThe model trains on the **Tiny Shakespeare** dataset — roughly 1.1 MB of text, about 1 million characters, drawn from the public-domain works of William Shakespeare. The vocabulary is character-level: 65 unique characters (letters, punctuation, newlines). This keeps the embedding table tiny and the whole training pipeline fast enough to run on a free Colab GPU.\n\nThe dataset is small enough to overfit in minutes, which makes it perfect for debugging. If your model cannot memorize Tiny Shakespeare, there is something wrong with your architecture or training loop.\n\nThe model is a **decoder-only Transformer** — the same family as GPT, LLaMA, and Mistral. It generates text autoregressively: given a sequence of characters, it predicts the next one, appends it to the context, and repeats.\n\nHere is the high-level architecture:\n\n*The decoder stack (right side of the original Transformer diagram) is the core of the model. We strip away the encoder entirely and keep only the masked self-attention and feed-forward blocks.*\n\n| Component | Value | \n|---|---|\n| Parameters | ~10.6M | \n| Layers ( `n_layer` ) | 6 | \n| Attention heads ( `n_head` ) | 6 | \n| Embedding dimension ( `n_embd` ) | 384 | \n| Context length ( `block_size` ) | 256 | \n| Feed-forward dimension | 4 × 384 = 1536 | \n| Dropout | 0.2 | \n| Vocabulary size | 65 | \n| Tokenization | Character-level | \n\nThese numbers are deliberately modest. A 6-layer, 6-head, 384-dimensional model is small enough to train on a free T4 in a couple of hours, yet large enough to produce coherent Shakespeare-like text after training.\n\nEach forward pass does the following:\n\nThe causal mask in the self-attention layer ensures that position `t` can only attend to positions `≤ t`. This is what makes the model autoregressive.\n\nThe entire training run fits comfortably within Google Colab's free T4 GPU (16 GB VRAM).\n\n| Setting | Value | \n|---|---|\n| Batch size | 64 | \n| Learning rate | 3e-4 (with cosine decay) | \n| Optimizer | AdamW | \n| Weight decay | 0.1 | \n| Warmup steps | 100 | \n| Max steps | 5,000 | \n| Gradient clipping | 1.0 | \n| Mixed precision | Yes (torch.cuda.amp) | \n\nA batch size of 64 with a context length of 256 means each forward pass processes 16,384 characters. With gradient accumulation, you can simulate larger batches if needed, but the free T4 handles this comfortably.\n\nTo fit within Colab's memory limits:\n\n`torch.cuda.amp.autocast`) halves the memory footprint of activations.\nTraining for 5,000 steps takes roughly 30–45 minutes on a T4. The loss curve typically drops from ~4.2 (random) to ~1.5–1.8, producing text that is recognizably Shakespearean in structure, if not in meaning.\n\nAfter training, the model generates text like this:\n\n```\nROMEO:\nWhat say'st thou, my lord? I will not be so:\nThe gentle heart is not a little word,\nAnd yet the world is grown so bad, that men\nDo call it virtue when they are most accursed.\n```\n\nIt is not Shakespeare. But it is Shakespeare-*shaped* — and that is the point. The model learned grammar, punctuation, character names, and the rhythm of Elizabethan dialogue purely from next-character prediction.\n\nThe [repository](https://github.com/Devn913/Tiny_Shakespeare_LLM_from_scratch) is organized for clarity:\n\n```\nTiny_Shakespeare_LLM_from_scratch/\n├── data/\n│   └── shakespeare.txt        # Tiny Shakespeare dataset\n├── model.py                   # Transformer architecture\n├── train.py                   # Training loop\n├── generate.py                # Inference script\n├── config.py                  # Hyperparameters\n└── README.md\n```\n\nEvery file is intentionally short and readable. The model definition is under 200 lines. The training loop is under 100.\n\nThis project is not about building the best language model. It is about building *a* language model — one you understand completely, from the embedding lookup to the final softmax.\n\nIf you want to run it yourself, clone the repository, open the notebook in Colab, and press play. The entire pipeline is designed to run on the free tier.\n\n**Repository:** [https://github.com/Devn913/Tiny_Shakespeare_LLM_from_scratch](https://github.com/Devn913/Tiny_Shakespeare_LLM_from_scratch)\n\n*Built with PyTorch on a free Google Colab T4.*", "url": "https://wpnews.pro/news/building-and-training-llm-from-scratch-no-gpu", "canonical_source": "https://dev.to/devn913/building-llm-from-scratch-no-gpu-1mc1", "published_at": "2026-09-21 07:05:17+00:00", "updated_at": "2026-09-21 07:23:26.363242+00:00", "lang": "en", "topics": ["large-language-models", "natural-language-processing", "ai-research", "neural-networks", "generative-ai"], "entities": ["Google Colab", "PyTorch", "Tiny Shakespeare", "Hugging Face", "GPT", "LLaMA", "Mistral", "GitHub"], "alternates": {"html": "https://wpnews.pro/news/building-and-training-llm-from-scratch-no-gpu", "markdown": "https://wpnews.pro/news/building-and-training-llm-from-scratch-no-gpu.md", "text": "https://wpnews.pro/news/building-and-training-llm-from-scratch-no-gpu.txt", "jsonld": "https://wpnews.pro/news/building-and-training-llm-from-scratch-no-gpu.jsonld"}}