{"slug": "transformers-understanding-the-architecture-behind-modern-ai", "title": "Transformers: Understanding the Architecture Behind Modern AI", "summary": "A developer explains the Transformer architecture, the foundation of modern AI models like ChatGPT, covering its encoder-decoder structure, positional encodings, and multi-head attention. The post details how transformers overcome limitations of earlier RNNs and attention mechanisms by using multi-head attention to handle long sequences effectively.", "body_md": "Transformers are the heart of modern AI models. AI has seen a lot of breakthrough advancements from ChatGPT to AI, and now. After the development of Transformers, translations and question-answering tasks have seen a breakthrough.\n\nIn this blog, I will try to explain Transformers from the basic concepts to how the complete architecture works.\n\nIt's not that Transformers were the first architecture to exist. Various architectures existed before, like RNNs, which take the current input and also the previous input.\n\nFor example:\n\nI am a boy\n\nt₁ = I\n\nt₂ = I + am\n\nt₃ = I + am + a\n\nt₄ = I + am + a + boy\n\nBut its popularity faded for long sequences as the effect of earlier inputs starts disappearing.\n\nThen the attention mechanism came, which focused not only on previous inputs but also on the relevant inputs to focus on. But their main limitation was that sentences with many words were not handled as effectively.\n\nTo overcome the above limitations, Transformers using multi-head attention came into the picture.\n\nA Transformer is like a human that first encodes a sentence, understands it, and then decodes it according to the required task.\n\nThe Transformer has two major parts:\n\nTRANSFORMER\n\n/ \\\n\n/ \\\n\nENCODER DECODER\n\n↓ ↓\n\nUnderstands Generates\n\nthe input output\n\nThe original Transformer architecture contains multiple encoder and decoder layers.\n\nThe original Transformer uses 6 encoder layers and 6 decoder layers.\n\nBefore reaching the main architecture, there are some prerequisites.\n\nFirst, the entire sentence is divided into tokens and then converted into embeddings.\n\nFor example:\n\nI am a boy\n\n↓\n\n[I, am, a, boy]\n\n↓\n\nToken IDs\n\n↓\n\nEmbedding vectors\n\nEach single token is a vector of a particular dimension, for example (768, 1024) depending on the model.\n\nThink of it like what a particular word means in mathematics in a coordinate system.\n\nSo instead of giving the Transformer raw words, we convert every word/token into a numerical vector.\n\nSince we are working with long sequences, the position of each token is very important.\n\nWhich token comes first or second can completely change the meaning of a sentence.\n\nTherefore, positional encodings are used to provide information about the position of each token.\n\nHow do they work?\n\nThe original Transformer uses sine and cosine functions.\n\nPE(pos, 2i) = sin(pos / 10000^(2i/d_model))\n\nPE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))\n\nFor example, let's take:\n\nSentence = \"I love AI\"\n\nd_model = 4\n\nFor position 0:\n\nPE(0) = [0, 1, 0, 1]\n\nFor position 1:\n\nPE(1) ≈ [0.8415, 0.5403, 0.0100, 0.99995]\n\nFor position 2:\n\nPE(2) ≈ [0.9093, -0.4161, 0.0200, 0.9998]\n\nNow suppose the embedding of \"love\" is:\n\nEmbedding(love) = [0.4, 0.3, 0.8, 0.2]\n\nSince \"love\" is at position 1:\n\nEmbedding = [0.4000, 0.3000, 0.8000, 0.2000]\n\nPE(1) = [0.8415, 0.5403, 0.0100, 0.99995]\n\n```\n            ↓ ADD\n```\n\nTransformer input = [1.2415, 0.8403, 0.8100, 1.19995]\n\nSo positional encoding basically tells the model:\n\nWhat is the token + Where is the token?\n\nNow comes the most important part: Attention.\n\nConsider the sentence:\n\nI am a boy\n\nEach token is projected into the coordinate system as a vector.\n\nIt is broken down into three parts by transformations:\n\nQuery (Q)\n\nKey (K)\n\nValue (V)\n\nThink of them like this:\n\nQuery\n\n\"What am I looking for?\"\n\nKey\n\n\"See, this is how I look in the coordinate system. Check how much I match with your Query.\"\n\nValue\n\n\"This is what my value in the system contains.\"\n\nWhat happens is that the Query vector for each word is multiplied with the Key vectors for each word using a dot product.\n\nThe result is divided by √dₖ because for large dimensions the dot product can become large, pushing the probability toward one side.\n\nThen Softmax is applied.\n\nAttention(Q,K,V) = softmax(QKᵀ / √dₖ)V\n\nThe Softmax gives the attention weights.\n\nThen these weights are multiplied with the corresponding Value vectors.\n\nThis process is repeated multiple times to capture semantic similarity between tokens. This is known as multi-head attention.\n\nLet's actually calculate a small example.\n\nConsider:\n\nI am a boy\n\nFor simplicity, let:\n\ndₖ = 2\n\nSuppose the vectors are:\n\nToken Q K V\n\nI [1,0] [1,1] [1,0]\n\nam [0,1] [1,0] [0,1]\n\na [1,1] [1,1] [1,1]\n\nboy [1,-1] [0,1] [0,2]\n\nLet's calculate attention for the last token \"boy\".\n\nStep 1: Calculate Q × Kᵀ\n\nFor \"boy\":\n\nQ(boy) = [1,-1]\n\nAgainst every Key:\n\n[1,-1] · [1,1] = 0\n\n[1,-1] · [1,0] = 1\n\n[1,-1] · [1,1] = 0\n\n[1,-1] · [0,1] = -1\n\nSo:\n\nQKᵀ = [0, 1, 0, -1]\n\nStep 2: Scale by √dₖ\n\nSince:\n\ndₖ = 2\n\n√dₖ = √2 ≈ 1.414\n\nTherefore:\n\n[0, 1, 0, -1] / 1.414\n\n≈ [0, 0.707, 0, -0.707]\n\nStep 3: Apply Softmax\n\nSoftmax([0, 0.707, 0, -0.707])\n\n≈ [0.157, 0.256, 0.157, 0.430]\n\nThese are the attention weights.\n\nSo \"boy\" pays different amounts of attention to each token.\n\nStep 4: Multiply by V\n\n0.157 × [1,0]\n\n+\n\n0.256 × [0,1]\n\n+\n\n0.157 × [1,1]\n\n+\n\n0.430 × [0,2]\n\nTherefore:\n\n≈ [0.314, 1.017]\n\nSo the output representation for \"boy\" becomes approximately:\n\n[0.314, 1.017]\n\nThe important idea is not the particular numbers, but the process:\n\nQ\n\n↓\n\nQKᵀ\n\n↓\n\nScale by √dₖ\n\n↓\n\nSoftmax\n\n↓\n\nAttention weights\n\n↓\n\nWeighted sum of V\n\n↓\n\nContextual representation\n\nInstead of one attention operation, we run multiple attention heads in parallel.\n\nEach head can learn different types of relationships.\n\nFor example, one head may focus on:\n\nsubject ↔ verb\n\nwhile another may focus on:\n\nverb ↔ object\n\nThe outputs of all heads are then concatenated and passed through a linear transformation.\n\nInput\n\n│\n\n┌───────────┼───────────┐\n\n↓ ↓ ↓\n\nHead 1 Head 2 Head 3 ... Head h\n\n↓ ↓ ↓\n\n└───────────┼───────────┘\n\n↓\n\nConcatenate\n\n↓\n\nLinear Layer\n\n↓\n\nOutput\n\nThe equation is:\n\nheadᵢ = Attention(QWᵢQ, KWᵢK, VWᵢV)\n\nMultiHead(Q,K,V) =\n\nConcat(head₁, head₂, ..., headₕ)Wᴼ\n\nAfter the attention operation, the output is combined with the original input using a residual connection.\n\nThen layer normalization is applied.\n\nThe basic idea is:\n\nInput\n\n↓\n\nAttention\n\n↓\n\nAdd original input\n\n↓\n\nLayer Normalization\n\n↓\n\nOutput\n\nResidual connections help preserve information and allow gradients to flow through deeper networks.\n\nLayer normalization helps stabilize the network during training.\n\nFurther, a Feed-Forward Neural Network (FFN) is applied to each token.\n\nThe equation is:\n\nFFN(x) = max(0, xW₁ + b₁)W₂ + b₂\n\nThe FFN is applied independently to every token.\n\nIts purpose is to further transform the information received from the self-attention layer and introduce non-linearity.\n\nSo a simple way to remember it is:\n\nAttention: Which tokens should interact?\n\nFFN: What transformation should be applied to the resulting representation?\n\nAfter the FFN, residual connection and layer normalization are applied again.\n\nThe above entire process forms one encoder layer.\n\nInput Embedding\n\n+\n\nPositional Encoding\n\n↓\n\nMulti-Head Self-Attention\n\n↓\n\nAdd & Layer Normalization\n\n↓\n\nFeed-Forward Network\n\n↓\n\nAdd & Layer Normalization\n\n↓\n\nEncoder Output\n\nThe original Transformer has 6 such encoder layers stacked together.\n\nEncoder Layer 1\n\n↓\n\nEncoder Layer 2\n\n↓\n\nEncoder Layer 3\n\n↓\n\nEncoder Layer 4\n\n↓\n\nEncoder Layer 5\n\n↓\n\nEncoder Layer 6\n\n↓\n\nFinal Encoder Output\n\nEach layer builds a richer contextual representation.\n\nAfter the encoder finishes, the output embedding is passed to the decoder.\n\nAfter applying positional embeddings, the decoder undergoes masked multi-head attention.\n\nUnlike the encoder, the decoder takes the output token by token at a time.\n\nFor example:\n\nAt the first step, the decoder predicts the first output token.\n\nThen:\n\n→ I\n\nThen:\n\nI → am\n\nThen:\n\nI am → a\n\nand so on.\n\nThis is called autoregressive generation.\n\nThe decoder must not see future tokens while generating the current token.\n\nFor example, if we are predicting:\n\nI am a boy\n\nwhile predicting \"a\", the decoder should not already know \"boy\".\n\nTherefore, a mask is applied.\n\nI am a boy\n\nI ✓\n\nam ✓ ✓\n\na ✓ ✓ ✓\n\nboy ✓ ✓ ✓ ✓\n\nThis is called causal/masked self-attention.\n\nThe target sequence is also right-shifted, meaning the decoder receives previously generated/known tokens to predict the next token.\n\nThis is an important part of the Transformer.\n\nThe final output of Encoder 6 is used by the cross-attention sublayer of the decoder layers.\n\nIn cross-attention:\n\nQuery (Q) → comes from decoder\n\nKey (K) → comes from final encoder output\n\nValue (V) → comes from final encoder output\n\nSo:\n\nEncoder\n\n↓\n\nEncoder Layer 6\n\n↓\n\nFinal Encoder Output\n\n│\n\n┌───────────┼───────────┐\n\n↓ ↓ ↓\n\nDecoder 1 Decoder 2 ... Decoder 6\n\nCross-Attn Cross-Attn Cross-Attn\n\nThe decoder's own output flows from one decoder layer to the next, while the encoder's final representation is available to the cross-attention of every decoder layer.\n\nA decoder layer therefore contains:\n\nTarget Embedding\n\n+\n\nPositional Encoding\n\n↓\n\nMasked Multi-Head Self-Attention\n\n↓\n\nAdd & LayerNorm\n\n↓\n\nMulti-Head Cross-Attention\n\n↑\n\nFinal Encoder Output\n\n↓\n\nAdd & LayerNorm\n\n↓\n\nFeed-Forward Network\n\n↓\n\nAdd & LayerNorm\n\n↓\n\nDecoder Output\n\nThe original Transformer contains 6 decoder layers.\n\nAfter passing through the decoder layers, a linear layer is applied.\n\nThen Softmax converts the final values into probabilities.\n\nDecoder Output\n\n↓\n\nLinear Layer\n\n↓\n\nSoftmax\n\n↓\n\nProbability of each token\n\n↓\n\nMost suitable next token\n\nFor example:\n\nI am a ______\n\nboy → 0.72\n\ngirl → 0.12\n\nstudent → 0.08\n\ndoctor → 0.03\n\n...\n\nThe model selects a token according to the generation strategy being used.\n\nHere is the complete flow of the original encoder-decoder Transformer:\n\nINPUT SENTENCE\n\n↓\n\nTOKENIZATION\n\n↓\n\nEMBEDDING\n\n↓\n\nPOSITIONAL ENCODING\n\n↓\n\n┌──────────────────────┐\n\n│ ENCODER × 6 │\n\n│ │\n\n│ Multi-Head Attention │\n\n│ ↓ │\n\n│ Add & Norm │\n\n│ ↓ │\n\n│ FFN │\n\n│ ↓ │\n\n│ Add & Norm │\n\n└──────────┬───────────┘\n\n↓\n\nFINAL ENCODER OUTPUT\n\n│\n\n│\n\n↓\n\nTARGET → Embedding + Positional Encoding\n\n↓\n\n┌──────────────────────┐\n\n│ DECODER × 6 │\n\n│ │\n\n│ Masked Self-Attention│\n\n│ ↓ │\n\n│ Add & Norm │\n\n│ ↓ │\n\n│ Cross-Attention ←┘\n\n│ ↓\n\n│ Add & Norm\n\n│ ↓\n\n│ FFN\n\n│ ↓\n\n│ Add & Norm\n\n└──────────┬───────────┘\n\n↓\n\nLinear Layer\n\n↓\n\nSoftmax\n\n↓\n\nOUTPUT TOKEN\n\n↓\n\nNEXT TOKEN ...\n\nThe Transformer can now be understood as a sequence of operations:\n\nWords\n\n↓\n\nTokens\n\n↓\n\nEmbeddings\n\n↓\n\nPositional Information\n\n↓\n\nSelf-Attention\n\n↓\n\nMulti-Head Attention\n\n↓\n\nFeed-Forward Network\n\n↓\n\nRepeat through encoder layers\n\n↓\n\nFinal Encoder Representation\n\n↓\n\nDecoder\n\n↓\n\nMasked Self-Attention\n\n↓\n\nCross-Attention with Encoder Output\n\n↓\n\nFeed-Forward Network\n\n↓\n\nRepeat through decoder layers\n\n↓\n\nLinear + Softmax\n\n↓\n\nOutput\n\nThe most important idea is that attention allows the model to decide which other tokens are relevant when building the representation of a particular token.\n\nTransformers have become extremely important in modern AI.\n\nSome applications include:\n\nMachine translation\n\nText generation\n\nQuestion answering\n\nText summarization\n\nSentiment analysis\n\nCode generation\n\nImage understanding\n\nMultimodal AI\n\nModels such as BERT, GPT and Vision Transformers are based on the Transformer idea, although their architectures can differ from the original encoder-decoder Transformer.\n\nTransformers changed the way we process sequential data by introducing attention as the central mechanism for understanding relationships between tokens.\n\nInstead of processing information strictly one token after another like traditional recurrent architectures, Transformers can use attention to determine which tokens are important to each other.\n\nThe major components are:\n\nTokenization\n\n↓\n\nEmbeddings\n\n↓\n\nPositional Encoding\n\n↓\n\nSelf-Attention\n\n↓\n\nMulti-Head Attention\n\n↓\n\nFeed-Forward Network\n\n↓\n\nResidual Connections + Layer Normalization\n\n↓\n\nEncoder / Decoder\n\n↓\n\nCross-Attention\n\n↓\n\nLinear + Softmax\n\n↓\n\nOutput\n\nUnderstanding these components makes it much easier to understand modern architectures such as BERT, GPT and other Transformer-based models.\n\nThis is how Transformers work — from converting words into vectors to using attention to understand relationships and finally generating an output.", "url": "https://wpnews.pro/news/transformers-understanding-the-architecture-behind-modern-ai", "canonical_source": "https://dev.to/arham_ahmed_63699c0d1def9/transformers-understanding-the-architecture-behind-modern-ai-537d", "published_at": "2026-08-28 19:32:46+00:00", "updated_at": "2026-08-28 19:48:11.160979+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "neural-networks"], "entities": ["ChatGPT"], "alternates": {"html": "https://wpnews.pro/news/transformers-understanding-the-architecture-behind-modern-ai", "markdown": "https://wpnews.pro/news/transformers-understanding-the-architecture-behind-modern-ai.md", "text": "https://wpnews.pro/news/transformers-understanding-the-architecture-behind-modern-ai.txt", "jsonld": "https://wpnews.pro/news/transformers-understanding-the-architecture-behind-modern-ai.jsonld"}}