{"slug": "understanding-transformers-by-building-one-from-scratch", "title": "Understanding Transformers by Building One from Scratch", "summary": "A developer built a Transformer model from scratch to translate English to Sanskrit, training a custom Byte Pair Encoding tokenizer with about 5,000 tokens and a model with roughly 22 million parameters on an RTX 3060 Laptop GPU with 6 GB of VRAM. The project aimed to demystify the architecture by implementing every component manually, including sinusoidal positional encodings and multi-head attention, leading to a deeper understanding of Transformers.", "body_md": "Over the past year, I built several applications around Large Language Models, from RAG systems to AI agents and multi-agent orchestration frameworks. While I understood how to use these models effectively, I still treated them as a black box. I couldn’t confidently explain what was happening inside one.\n\nThis wasn’t the first time I had faced that problem. Earlier, implementing a neural network and Proximal Policy Optimization (PPO) from scratch taught me that building something yourself leads to a much deeper understanding than simply using it. I wanted to approach Transformers the same way.\n\nThe turning point came when I revisited the architecture diagram from [ Attention Is All You Need](https://arxiv.org/pdf/1706.03762). I recognized the components, but didn’t understand why they existed or how they fit together. My goal became simple: build the architecture from scratch until that diagram no longer felt intimidating.\n\nInstead of another decoder-only language model, I chose an English-to-Sanskrit translation task using the [ Sāmayik](https://huggingface.co/papers/2305.14004) dataset. It gave me the opportunity to implement the complete encoder-decoder Transformer while working on a language I had always wanted to explore.\n\nLike most people learning Transformers, I initially wanted to jump straight into attention. It didn’t take long before I realized I was skipping one of the most fundamental parts of the pipeline: how text is represented.\n\nInstead of using an existing tokenizer, I trained my own Byte Pair Encoding (BPE) tokenizer and experimented with different vocabulary sizes before settling on about **5,000 tokens**.\n\nBuilding the tokenizer completely changed how I viewed the problem. A Transformer never actually sees English or Sanskrit. It only sees sequences of token IDs, which are eventually converted into vectors. Until then, I had always thought of tokenization as preprocessing. By the end of implementing it, I realized it was one of the first architectural decisions of the entire system.\n\nI had to be mindful of the fact that I was on a laptop with an RTX 3060 Laptop GPU and only **6 GB of VRAM** which meant that every decision mattered. Vocabulary size, embedding dimensions, sequence length, and eventually the model architecture itself all had to fit within those constraints.\n\nWith the tokenizer complete, I finally had a way of representing language numerically. The next step was assembling the Transformer itself.\n\nWith the tokenizer complete, I could finally start building the Transformer.\n\nRather than relying on torch.nn.Transformer, every major component was implemented manually. The final model consisted of learned source and target embeddings, sinusoidal positional encodings, multi-head attention, encoder and decoder stacks, feed-forward networks, residual connections, layer normalization, masking, teacher forcing during training, and greedy autoregressive decoding during inference. In total, the model contained roughly **22 million trainable parameters**.\n\nAs I implemented each component, I gradually stopped seeing them as individual blocks and started seeing them as solutions to specific problems. That shift turned out to be far more valuable than memorizing the architecture itself.\n\nFor example, I chose the original sinusoidal positional encoding instead of learned positional embeddings. Initially, this was simply because it introduced no additional trainable parameters and fit within my hardware constraints. Later, I realized that the sine and cosine functions weren’t the important part. They were just one way of introducing positional information. Once that clicked, understanding why modern architectures use alternatives like RoPE became much easier.\n\nThe same thing happened with attention. Query, Key, and Value initially felt like three confusing matrices. After implementing scaled dot-product attention, I stopped thinking about the matrices and started thinking about the idea behind them. Every token is simply deciding **where to look** and **what information to retrieve** from the rest of the sequence.\n\nWorking with tensors also changed the way I approached debugging. Early on, I spent a lot of time trying to remember which matrices should multiply together. Eventually, I stopped thinking about the operations themselves and instead focused on the shape I wanted each block to produce. Once the expected input and output dimensions were clear, the intermediate tensor operations became much easier to reason about.\n\n``` python\nclass MultiHeadAttention(nn.Module):    def __init__(self, embedding_dims, weight_dim, num_heads):        super().__init__()        self.attention_heads = nn.ModuleList()        for _ in range(num_heads):            self.attention_heads.append(SingleHeadAttention(embedding_dims, weight_dim))        self.W_O = nn.Linear(weight_dim * num_heads, embedding_dims) # weight_dim is multiplied with num_heads, because this is going to be matrix multiplied against multiple concatenated heads            def forward(self, query_embeddings, key_embeddings, value_embeddings, mask=None):        head_outputs = [] # List of tensors, each will have shape (16 * 64 * weight_dim of SingleHeadAttention) or (batch_size * max_len * weight_dim of SingleHeadAttention)        for head in self.attention_heads:            head_outputs.append(head(query_embeddings, key_embeddings, value_embeddings, mask))                multi_head_output = torch.cat(head_outputs, dim=-1) # Each head outputs (batch_size, seq_len, weight_dim). Concatenating along the last dimension gives (batch_size, seq_len, weight_dim * num_heads).        output = self.W_O(multi_head_output) # (16 * 64 * embedding_dims)                return output\n```\n\nPerhaps the biggest realization during implementation was that the original paper doesn’t describe *the* Transformer. It describes *a* Transformer. Modern architectures may replace positional encodings, attention mechanisms, or normalization layers, but the underlying ideas remain remarkably consistent. Understanding those ideas made reading newer papers far less intimidating.\n\nThe biggest conceptual breakthrough came while implementing training and inference.\n\nUntil this project, most neural networks I had worked with simply mapped inputs to outputs. Transformers do the same, but the way they are trained is quite different from how they generate text.\n\nDuring training, the decoder receives the correct target sentence shifted by one position. Combined with causal masking, this allows every token to be predicted in parallel. During inference, those target words don’t exist. The model has to generate one token at a time, feed it back into the decoder, and repeat until it predicts an end-of-sequence token.\n\nThis difference confused me for quite some time because the two procedures look completely different. Implementing both finally made the connection clear. The model is always learning to predict **the next token**. Teacher forcing simply allows that objective to be trained efficiently in parallel.\n\n```\ntokenizer = BPETokenizer(tok5000[\"merges\"], vocab)CHECKPOINT = \"checkpoints/dropout/best_transformer_checkpoint.pth\"checkpoint = torch.load(CHECKPOINT, map_location=device)cfg = checkpoint[\"config\"]model = Transformer(embedding_dims=cfg['embedding_dims'], max_len=cfg[\"max_len\"], vocab_size=cfg[\"vocab_size\"], weight_dim=cfg[\"weight_dim\"], num_heads=cfg[\"num_heads\"], ff_dims=cfg[\"ff_dims\"], num_encoder_blocks=cfg[\"num_encoder_blocks\"], num_decoder_blocks=cfg[\"num_decoder_blocks\"],).to(device)model.load_state_dict(checkpoint[\"model_state_dict\"])model.eval()def translate(sentence, id_printing=True):    src = tokenizer.encode(sentence)    src = src[:cfg[\"max_len\"]-1] # clipping len of input to max allowed len-1     src.append(EOS)    src = pad_sequence(PAD,EOS,src,cfg[\"max_len\"])    src = torch.tensor(src, dtype=torch.long).unsqueeze(0).to(device)    generated = [BOS]    with torch.no_grad():        for _ in range(cfg[\"max_len\"]):            tgt = torch.tensor(generated, dtype=torch.long).unsqueeze(0).to(device)            logits = model(src, tgt)            next_token = torch.argmax(logits[:,-1,:],dim=1).item()            generated.append(next_token)            if next_token == EOS:                break        if id_printing:            print(\"Generated IDs: \",generated)        output_tokens = [token for token in generated if token not in (PAD, BOS, EOS)]        return tokenizer.decode(output_tokens)\n```\n\nThat was probably the point where the paper stopped feeling like a collection of equations and started feeling like a carefully engineered system.\n\nWith the architecture complete, it was finally time to see whether the model could learn.\n\nThe final model contained roughly **22 million trainable parameters** and was trained on the **Sāmayik** English-Sanskrit dataset using teacher forcing and validation-based checkpointing. Everything was trained on my RTX 3060 Laptop GPU and a full training run took over **eight hours**.\n\nAs the project evolved, so did the training pipeline. What started as a simple training loop gradually gained validation, checkpointing, dropout, and the ability to resume training from saved checkpoints.\n\nI still remember waking up the next morning, excited to see how the model had trained overnight, only to realize that the validation loss had already started increasing while the training loss continued to decrease. It was a textbook case of overfitting.\n\nThe final model is far from a production-ready translator. On the training set, many translations were surprisingly accurate, which also confirmed that the model had overfit. On unseen examples, however, the quality dropped considerably. The generated Sanskrit was often grammatically incorrect or drifted away from the intended meaning.\n\nEven then, it was obvious that the model had learned something meaningful. It frequently identified the subject, captured actions such as reading or climbing, and often preserved the overall intent of the English sentence. The translations weren’t always correct, but they rarely felt completely random.\n\nOne observation I found particularly interesting came from using a **shared tokenizer** for both English and Sanskrit. Since the tokenizer was trained jointly, there was nothing inherently preventing the decoder from generating English tokens. The only thing encouraging Sanskrit generation was the target data it had seen during training.\n\nDuring inference, entering a single character like **“B”** produced **“Bank Accounts”**, which looked completely bizarre at first.\n\nAfter thinking about it, the behavior made perfect sense. Some target sentences in the dataset contained English words inside quotation marks, meaning the decoder had occasionally seen English tokens during training. With a shared vocabulary, generating English wasn’t impossible, just uncommon. It was a small detail, but it gave me another glimpse into what the model had actually learned.\n\nLooking back, I don’t think the biggest outcome of this project was learning how to implement a Transformer. It was learning how to think about one.\n\nThe original paper no longer feels like a collection of equations. Instead, I see a sequence of design decisions, each solving a particular problem. I also stopped trying to memorize tensor operations and began thinking about the shape each block should produce, making the implementation much easier to reason about.\n\nThere are still many parts of modern language models that I don’t fully understand, but I no longer find new architectures intimidating. Instead of asking *“How does this work?”*, I now find myself asking *“Why was this design chosen, and what problem is it solving?”*\n\nThat shift in perspective was the most valuable outcome of this project.\n\nIf you’re on a similar journey of understanding Transformers from first principles, I hope this project helps. The complete implementation is available on [GitHub](https://github.com/jhawaritvik/Eng2Sans-Transformer).\n\n[Understanding Transformers by Building One from Scratch](https://pub.towardsai.net/understanding-transformers-by-building-one-from-scratch-6a0dfe3d146d) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.", "url": "https://wpnews.pro/news/understanding-transformers-by-building-one-from-scratch", "canonical_source": "https://pub.towardsai.net/understanding-transformers-by-building-one-from-scratch-6a0dfe3d146d?source=rss----98111c9905da---4", "published_at": "2026-08-05 12:01:59+00:00", "updated_at": "2026-08-05 12:22:40.034219+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "natural-language-processing"], "entities": ["Transformer", "Byte Pair Encoding", "RTX 3060 Laptop GPU", "Sāmayik", "Attention Is All You Need"], "alternates": {"html": "https://wpnews.pro/news/understanding-transformers-by-building-one-from-scratch", "markdown": "https://wpnews.pro/news/understanding-transformers-by-building-one-from-scratch.md", "text": "https://wpnews.pro/news/understanding-transformers-by-building-one-from-scratch.txt", "jsonld": "https://wpnews.pro/news/understanding-transformers-by-building-one-from-scratch.jsonld"}}