{"slug": "i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about", "title": "I Built a Small LLM From Scratch — Here’s What I Finally Understood About Transformer", "summary": "A developer built a small character-level language model from scratch in PyTorch, implementing a transformer pipeline that includes RMSNorm, Grouped Query Attention with RoPE, and a SwiGLU feed-forward network. The model, configured with a 65-token vocabulary, 256-dimensional embeddings, four transformer blocks, and a 256-token context length, was trained on Shakespeare text to predict the next character. The developer said the exercise was meant to move past terminology, noting that \"before a model can learn language, language has to become numerical data.", "body_md": "I used to understand Large Language Models mostly from the outside.\n\nI knew terms like:\n\nBut knowing the terminology is very different from understanding what actually happens inside a model.\n\nSo instead of only reading about LLMs, I decided to **build a small language model from scratch using PyTorch**.\n\nThe goal wasn't to build the next ChatGPT.\n\nThe goal was much more important for me as a learner:\n\nThe model starts with Shakespeare text and learns to predict the next character.\n\nThe basic pipeline is:\n\n```\nText\n  ↓\nCharacter Tokenization\n  ↓\nToken Embeddings\n  ↓\nTransformer Blocks\n  ↓\nRMSNorm\n  ↓\nGrouped Query Attention + RoPE\n  ↓\nSwiGLU Feed Forward Network\n  ↓\nFinal Normalization\n  ↓\nLanguage Model Head\n  ↓\nNext-token prediction\n```\n\nThe model configuration I worked with was:\n\n```\nVocabulary size:       65\nEmbedding dimension:   256\nTransformer blocks:     4\nQuery heads:            8\nKV heads:               2\nSwiGLU hidden size:     680\nContext length:         256\n```\n\nThe important thing is that these aren't just names in a diagram. I implemented the pieces and followed the tensors through the model.\n\nNeural networks don't understand strings directly.\n\nSo the first step was tokenization.\n\nFor this project, I used a **character-level tokenizer**.\n\nFor example:\n\n```\n\"Hello\"\n```\n\ncan become something like:\n\n```\n[20, 43, 50, 50, 53]\n```\n\nThe tokenizer maintains two mappings:\n\n```\nchar_to_idx = {\n    character: integer\n}\n\nidx_to_char = {\n    integer: character\n}\n```\n\nThen:\n\n``` python\ndef encode(s):\n    return [char_to_idx[c] for c in s]\n\ndef decode(ids):\n    return \"\".join([idx_to_char[i] for i in ids])\n```\n\nThe important lesson for me was simple:\n\n**Before a model can learn language, language has to become numerical data.**\n\nThe training examples are created by shifting the sequence by one position.\n\n```\nInput:  To be o\nTarget: o be or\n```\n\nSo the model is continuously learning:\n\n```\nGiven previous characters → predict the next character\n```\n\nThis turns one sequence of length `N` into `N` next-token prediction examples.\n\nOne component I learned about was **RMSNorm (Root Mean Square Normalization)**.\n\nTraditional LayerNorm performs operations involving the mean and standard deviation.\n\nRMSNorm simplifies this.\n\nConceptually:\n\n```\nx\n ↓\ncalculate RMS\n ↓\nnormalize x\n ↓\nlearnable scale\n```\n\nThe implementation is roughly:\n\n```\nrms = torch.sqrt(\n    x.pow(2).mean(dim=-1, keepdim=True) + eps\n)\n\noutput = (x / rms) * weight\n```\n\nThe interesting part is that RMSNorm only needs a learnable scale vector.\n\nMy takeaway:\n\n**Normalization isn't just a mathematical trick. It helps keep the values flowing through a deep network under control.**\n\nThe material compares RMSNorm with LayerNorm and notes its use in modern architectures such as LLaMA, Mistral, and Gemma.\n\nThis was one of the concepts I found most interesting.\n\nA transformer needs positional information.\n\nOne approach is to add positional embeddings.\n\nRoPE — **Rotary Positional Embeddings** — takes a different approach.\n\nInstead of simply adding a position vector, RoPE rotates the Query and Key vectors.\n\nThe important intuition is:\n\n```\nPosition 3 → Position 5\ndistance = 2\n\nPosition 7 → Position 9\ndistance = 2\n```\n\nRoPE allows the attention mechanism to capture this kind of **relative positional relationship**.\n\nAt a high level:\n\n```\nQuery + position\nKey + position\n       ↓\nrotations\n       ↓\nattention\n```\n\nThe rotation is applied to pairs of dimensions:\n\n```\n(d0, d1)\n(d2, d3)\n(d4, d5)\n...\n```\n\nand each pair gets its own rotation frequency.\n\nThe biggest lesson I took from this:\n\n**Position isn't just “where is this token?” — attention can benefit from understanding how far apart tokens are.**\n\nNext came **Grouped Query Attention (GQA)**.\n\nThis was especially useful because it showed me that modern LLM architecture isn't only about making models bigger.\n\nIt's also about making them **more efficient**.\n\nInstead of giving every Query head its own Key and Value heads, GQA allows multiple Query heads to share KV heads.\n\nIn my model:\n\n```\nQuery heads = 8\nKV heads    = 2\n```\n\nSo four Query heads share each KV head.\n\n```\nQ0 ─┐\nQ1 ─┤\nQ2 ─┤──→ K0, V0\nQ3 ─┘\n\nQ4 ─┐\nQ5 ─┤\nQ6 ─┤──→ K1, V1\nQ7 ─┘\n```\n\nThis significantly reduces the size of the KV cache.\n\nThe implementation first projects Q, K and V, reshapes them into heads, applies RoPE to Q and K, and then expands the KV heads to match the Query heads.\n\nThis taught me an important engineering lesson:\n\n**AI engineering isn't only about model accuracy. Memory usage and inference efficiency matter too.**\n\nDuring training, the model shouldn't be allowed to cheat.\n\nSuppose we're predicting:\n\n```\nThe cat sat\n```\n\nWhen predicting the next token after `\"The\"`, the model shouldn't be able to look at `\"cat\"`.\n\nThat's why causal masking is used.\n\n```\n       T0 T1 T2 T3\n\nT0     ✓  ✗  ✗  ✗\nT1     ✓  ✓  ✗  ✗\nT2     ✓  ✓  ✓  ✗\nT3     ✓  ✓  ✓  ✓\n```\n\nEach token can attend to itself and previous tokens, but not future tokens.\n\nThe attention computation looks roughly like:\n\n```\nscores = (q @ k.transpose(-2, -1)) * scale\n\nscores = scores.masked_fill(\n    mask,\n    float(\"-inf\")\n)\n\nweights = F.softmax(scores, dim=-1)\n\nout = weights @ v\n```\n\nThis made the phrase **“next-token prediction”** much more concrete for me.\n\nAnother component I implemented was **SwiGLU**.\n\nA traditional feed-forward network can look like:\n\n```\nInput\n  ↓\nLinear\n  ↓\nReLU\n  ↓\nLinear\n  ↓\nOutput\n```\n\nSwiGLU introduces a gating mechanism.\n\nIt has two paths:\n\n```\n             ┌──→ SiLU ──┐\nInput ───────┤            × ──→ Linear ──→ Output\n             └──→ Up ────┘\n```\n\nThe core operation is:\n\n```\ngate = F.silu(self.w_gate(x))\nup = self.w_up(x)\n\noutput = self.w_down(gate * up)\n```\n\nThe gate learns which information should be emphasized or suppressed.\n\nThe training material describes this as a gated dual-path feed-forward network, with SiLU on the gate path and element-wise multiplication controlling information flow.\n\nA transformer block in my implementation follows this pattern:\n\n```\nInput\n  │\n  ├── RMSNorm\n  │\n  ├── GQA + RoPE\n  │\n  └── Residual Add\n          ↓\n        x'\n          │\n          ├── RMSNorm\n          │\n          ├── SwiGLU\n          │\n          └── Residual Add\n                 ↓\n               Output\n```\n\nIn code:\n\n```\nx = x + self.attention(\n    self.attn_norm(x),\n    rope_cos,\n    rope_sin\n)\n\nx = x + self.ffn(\n    self.ffn_norm(x)\n)\n```\n\nThis is a **pre-norm + residual** architecture.\n\nThe residual connection means the block learns a change to the existing representation instead of having to completely rebuild it at every layer.\n\nThe complete model then becomes:\n\n```\nCharacter IDs\n     ↓\nToken Embedding\n     ↓\nTransformer Block × 4\n     │\n     ├── RMSNorm\n     ├── GQA\n     │    └── RoPE\n     ├── Residual\n     ├── RMSNorm\n     ├── SwiGLU\n     └── Residual\n     ↓\nFinal RMSNorm\n     ↓\nLanguage Model Head\n     ↓\nLogits\n     ↓\nSoftmax\n     ↓\nNext Character\n```\n\nThe model uses weight tying between the token embedding and language-model head.\n\nThe complete architecture in the project uses 4 transformer blocks, 8 Query heads, 2 KV heads, and a 256-dimensional representation.\n\nOnce the architecture was ready, the next question was:\n\n**How does it actually learn?**\n\nThe training loop follows the familiar deep-learning pattern:\n\n```\nGet batch\n   ↓\nForward pass\n   ↓\nCalculate loss\n   ↓\nZero gradients\n   ↓\nBackward pass\n   ↓\nGradient clipping\n   ↓\nOptimizer step\n```\n\nThe project used:\n\n```\nBatch size:      64\nLearning rate:   3e-4\nOptimizer:       AdamW\nTraining steps:  3000\n```\n\nThe training code follows:\n\n```\nxb, yb = get_batch(...)\n\nlogits, loss = model(xb, yb)\n\noptimizer.zero_grad()\n\nloss.backward()\n\ntorch.nn.utils.clip_grad_norm_(\n    model.parameters(),\n    max_norm=1.0\n)\n\noptimizer.step()\n```\n\nGradient clipping is used to reduce the risk of exploding gradients during training.\n\nOne thing I don't want to overlook when building ML models is evaluation.\n\nA model getting better on training data doesn't automatically mean it's generalizing well.\n\nSo we can compare:\n\n```\nTraining loss\n      vs\nValidation loss\n```\n\nA growing gap between them can be a signal of overfitting.\n\nDuring evaluation, dropout is disabled and gradients aren't needed:\n\n``` python\n@torch.no_grad()\ndef estimate_loss():\n    model.eval()\n    ...\n    model.train()\n```\n\nThe training material uses the train/validation loss gap as an indicator for overfitting.\n\nAfter training, the model can generate new text.\n\nThe process is autoregressive:\n\n```\nPrompt\n  ↓\nModel\n  ↓\nPredict next token\n  ↓\nAppend token\n  ↓\nModel again\n  ↓\nPredict next token\n  ↓\nRepeat...\n\"ROMEO:\"\n```\n\ngets encoded into token IDs.\n\nThe model predicts the next character, adds it to the sequence, and repeats the process.\n\nThe generation code takes the logits from the **last position**, converts them into probabilities using softmax, samples the next token, and appends it to the sequence.\n\nAnother concept that became much clearer after implementing generation was **temperature**.\n\nLower temperature:\n\n```\nMore predictable\nLess random\n```\n\nHigher temperature:\n\n```\nMore diverse\nMore random\nT = 0.3 → conservative\nT = 0.8 → balanced\nT = 1.5 → creative\n```\n\nSo temperature doesn't make the model “smarter”.\n\nIt changes how we sample from its probability distribution.\n\nThe biggest lesson wasn't memorizing what RMSNorm, RoPE, GQA and SwiGLU stand for.\n\nIt was understanding how the pieces connect.\n\nBefore this project, I could say:\n\n“Transformers use attention.”\n\nNow I can explain:\n\n```\nText\n ↓\nTokens\n ↓\nEmbeddings\n ↓\nQ / K / V\n ↓\nRoPE\n ↓\nCausal Attention\n ↓\nGQA\n ↓\nResidual Connection\n ↓\nSwiGLU\n ↓\nRepeated Transformer Blocks\n ↓\nLogits\n ↓\nProbability Distribution\n ↓\nNext Token\n```\n\nThat difference is important.\n\n**Knowing the vocabulary is not the same as understanding the system.**\n\nAs someone preparing for an AI/ML engineering role, I think there is value in going one layer deeper.\n\nUsing:\n\n```\nresponse = client.chat(...)\n```\n\nis useful.\n\nBut understanding what happens underneath gives me a stronger foundation for:\n\nI don't expect a small Shakespeare model to qualify me for an LLM engineering role by itself.\n\nBut building it gave me something more valuable:\n\n**A mental model of how modern language models work.**\n\nReading about transformers is useful.\n\nImplementing one forces you to understand the tensor shapes and data flow.\n\nDon't memorize:\n\n```\nRoPE\nGQA\nRMSNorm\nSwiGLU\n```\n\nAsk:\n\n```\nWhy is it there?\nWhat problem does it solve?\nWhat happens if I remove it?\n```\n\nOne of the best ways I found to understand deep learning code is to constantly ask:\n\n```\nWhat is the shape here?\n[b, seq, 256]\n      ↓\n[b, 8, seq, 32]\n      ↓\nattention\n      ↓\n[b, seq, 256]\n```\n\nA good ML engineer needs to think about:\n\n```\nData\n↓\nArchitecture\n↓\nTraining\n↓\nEvaluation\n↓\nInference\n↓\nEfficiency\n```\n\nThis is probably the biggest career lesson I took from this.\n\nDon't just put:\n\n“Built an LLM using PyTorch”\n\non a resume.\n\nBe able to explain:\n\n“I implemented character tokenization, causal next-token prediction, RMSNorm, RoPE, GQA with shared KV heads, SwiGLU, residual transformer blocks, AdamW training, validation loss evaluation, and autoregressive generation.”\n\nThat demonstrates actual understanding.\n\nThis project gave me a foundation.\n\nMy next step is to move from:\n\n**“I understand how the architecture works.”**\n\nto:\n\n**“I can build useful AI systems with it.”**\n\nThat means exploring things like:\n\nThe goal isn't to learn every AI buzzword.\n\nThe goal is to **build things, understand them, and document what I learn.**\n\nI'm learning that the fastest way to improve as an AI/ML engineer isn't to wait until I know everything.\n\nIt's to:\n\n**Learn → Build → Break → Debug → Explain → Share → Repeat.**\n\nThis small LLM project was one step in that process.\n\nAnd I'm going to keep documenting the journey.\n\n**Tech Stack:** Python · PyTorch · Deep Learning · Transformers · LLMs\n\n**What would you recommend I build next — a RAG system, fine-tuned model, or an AI agent?**", "url": "https://wpnews.pro/news/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about", "canonical_source": "https://dev.to/onlydev75/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about-transformer-2h2o", "published_at": "2026-10-21 05:13:19+00:00", "updated_at": "2026-10-21 05:17:27.167580+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "natural-language-processing"], "entities": ["PyTorch", "LLaMA", "Mistral", "Gemma"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about", "markdown": "https://wpnews.pro/news/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about.md", "text": "https://wpnews.pro/news/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about.txt", "jsonld": "https://wpnews.pro/news/i-built-a-small-llm-from-scratch-heres-what-i-finally-understood-about.jsonld"}}