# Transformers: Understanding the Architecture Behind Modern AI > Source: > Published: 2026-08-28 19:32:46+00:00 Transformers are the heart of modern AI models. AI has seen a lot of breakthrough advancements from ChatGPT to AI, and now. After the development of Transformers, translations and question-answering tasks have seen a breakthrough. In this blog, I will try to explain Transformers from the basic concepts to how the complete architecture works. It's not that Transformers were the first architecture to exist. Various architectures existed before, like RNNs, which take the current input and also the previous input. For example: I am a boy t₁ = I t₂ = I + am t₃ = I + am + a t₄ = I + am + a + boy But its popularity faded for long sequences as the effect of earlier inputs starts disappearing. Then the attention mechanism came, which focused not only on previous inputs but also on the relevant inputs to focus on. But their main limitation was that sentences with many words were not handled as effectively. To overcome the above limitations, Transformers using multi-head attention came into the picture. A Transformer is like a human that first encodes a sentence, understands it, and then decodes it according to the required task. The Transformer has two major parts: TRANSFORMER / \ / \ ENCODER DECODER ↓ ↓ Understands Generates the input output The original Transformer architecture contains multiple encoder and decoder layers. The original Transformer uses 6 encoder layers and 6 decoder layers. Before reaching the main architecture, there are some prerequisites. First, the entire sentence is divided into tokens and then converted into embeddings. For example: I am a boy ↓ [I, am, a, boy] ↓ Token IDs ↓ Embedding vectors Each single token is a vector of a particular dimension, for example (768, 1024) depending on the model. Think of it like what a particular word means in mathematics in a coordinate system. So instead of giving the Transformer raw words, we convert every word/token into a numerical vector. Since we are working with long sequences, the position of each token is very important. Which token comes first or second can completely change the meaning of a sentence. Therefore, positional encodings are used to provide information about the position of each token. How do they work? The original Transformer uses sine and cosine functions. PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) For example, let's take: Sentence = "I love AI" d_model = 4 For position 0: PE(0) = [0, 1, 0, 1] For position 1: PE(1) ≈ [0.8415, 0.5403, 0.0100, 0.99995] For position 2: PE(2) ≈ [0.9093, -0.4161, 0.0200, 0.9998] Now suppose the embedding of "love" is: Embedding(love) = [0.4, 0.3, 0.8, 0.2] Since "love" is at position 1: Embedding = [0.4000, 0.3000, 0.8000, 0.2000] PE(1) = [0.8415, 0.5403, 0.0100, 0.99995] ``` ↓ ADD ``` Transformer input = [1.2415, 0.8403, 0.8100, 1.19995] So positional encoding basically tells the model: What is the token + Where is the token? Now comes the most important part: Attention. Consider the sentence: I am a boy Each token is projected into the coordinate system as a vector. It is broken down into three parts by transformations: Query (Q) Key (K) Value (V) Think of them like this: Query "What am I looking for?" Key "See, this is how I look in the coordinate system. Check how much I match with your Query." Value "This is what my value in the system contains." What happens is that the Query vector for each word is multiplied with the Key vectors for each word using a dot product. The result is divided by √dₖ because for large dimensions the dot product can become large, pushing the probability toward one side. Then Softmax is applied. Attention(Q,K,V) = softmax(QKᵀ / √dₖ)V The Softmax gives the attention weights. Then these weights are multiplied with the corresponding Value vectors. This process is repeated multiple times to capture semantic similarity between tokens. This is known as multi-head attention. Let's actually calculate a small example. Consider: I am a boy For simplicity, let: dₖ = 2 Suppose the vectors are: Token Q K V I [1,0] [1,1] [1,0] am [0,1] [1,0] [0,1] a [1,1] [1,1] [1,1] boy [1,-1] [0,1] [0,2] Let's calculate attention for the last token "boy". Step 1: Calculate Q × Kᵀ For "boy": Q(boy) = [1,-1] Against every Key: [1,-1] · [1,1] = 0 [1,-1] · [1,0] = 1 [1,-1] · [1,1] = 0 [1,-1] · [0,1] = -1 So: QKᵀ = [0, 1, 0, -1] Step 2: Scale by √dₖ Since: dₖ = 2 √dₖ = √2 ≈ 1.414 Therefore: [0, 1, 0, -1] / 1.414 ≈ [0, 0.707, 0, -0.707] Step 3: Apply Softmax Softmax([0, 0.707, 0, -0.707]) ≈ [0.157, 0.256, 0.157, 0.430] These are the attention weights. So "boy" pays different amounts of attention to each token. Step 4: Multiply by V 0.157 × [1,0] + 0.256 × [0,1] + 0.157 × [1,1] + 0.430 × [0,2] Therefore: ≈ [0.314, 1.017] So the output representation for "boy" becomes approximately: [0.314, 1.017] The important idea is not the particular numbers, but the process: Q ↓ QKᵀ ↓ Scale by √dₖ ↓ Softmax ↓ Attention weights ↓ Weighted sum of V ↓ Contextual representation Instead of one attention operation, we run multiple attention heads in parallel. Each head can learn different types of relationships. For example, one head may focus on: subject ↔ verb while another may focus on: verb ↔ object The outputs of all heads are then concatenated and passed through a linear transformation. Input │ ┌───────────┼───────────┐ ↓ ↓ ↓ Head 1 Head 2 Head 3 ... Head h ↓ ↓ ↓ └───────────┼───────────┘ ↓ Concatenate ↓ Linear Layer ↓ Output The equation is: headᵢ = Attention(QWᵢQ, KWᵢK, VWᵢV) MultiHead(Q,K,V) = Concat(head₁, head₂, ..., headₕ)Wᴼ After the attention operation, the output is combined with the original input using a residual connection. Then layer normalization is applied. The basic idea is: Input ↓ Attention ↓ Add original input ↓ Layer Normalization ↓ Output Residual connections help preserve information and allow gradients to flow through deeper networks. Layer normalization helps stabilize the network during training. Further, a Feed-Forward Neural Network (FFN) is applied to each token. The equation is: FFN(x) = max(0, xW₁ + b₁)W₂ + b₂ The FFN is applied independently to every token. Its purpose is to further transform the information received from the self-attention layer and introduce non-linearity. So a simple way to remember it is: Attention: Which tokens should interact? FFN: What transformation should be applied to the resulting representation? After the FFN, residual connection and layer normalization are applied again. The above entire process forms one encoder layer. Input Embedding + Positional Encoding ↓ Multi-Head Self-Attention ↓ Add & Layer Normalization ↓ Feed-Forward Network ↓ Add & Layer Normalization ↓ Encoder Output The original Transformer has 6 such encoder layers stacked together. Encoder Layer 1 ↓ Encoder Layer 2 ↓ Encoder Layer 3 ↓ Encoder Layer 4 ↓ Encoder Layer 5 ↓ Encoder Layer 6 ↓ Final Encoder Output Each layer builds a richer contextual representation. After the encoder finishes, the output embedding is passed to the decoder. After applying positional embeddings, the decoder undergoes masked multi-head attention. Unlike the encoder, the decoder takes the output token by token at a time. For example: At the first step, the decoder predicts the first output token. Then: → I Then: I → am Then: I am → a and so on. This is called autoregressive generation. The decoder must not see future tokens while generating the current token. For example, if we are predicting: I am a boy while predicting "a", the decoder should not already know "boy". Therefore, a mask is applied. I am a boy I ✓ am ✓ ✓ a ✓ ✓ ✓ boy ✓ ✓ ✓ ✓ This is called causal/masked self-attention. The target sequence is also right-shifted, meaning the decoder receives previously generated/known tokens to predict the next token. This is an important part of the Transformer. The final output of Encoder 6 is used by the cross-attention sublayer of the decoder layers. In cross-attention: Query (Q) → comes from decoder Key (K) → comes from final encoder output Value (V) → comes from final encoder output So: Encoder ↓ Encoder Layer 6 ↓ Final Encoder Output │ ┌───────────┼───────────┐ ↓ ↓ ↓ Decoder 1 Decoder 2 ... Decoder 6 Cross-Attn Cross-Attn Cross-Attn The decoder's own output flows from one decoder layer to the next, while the encoder's final representation is available to the cross-attention of every decoder layer. A decoder layer therefore contains: Target Embedding + Positional Encoding ↓ Masked Multi-Head Self-Attention ↓ Add & LayerNorm ↓ Multi-Head Cross-Attention ↑ Final Encoder Output ↓ Add & LayerNorm ↓ Feed-Forward Network ↓ Add & LayerNorm ↓ Decoder Output The original Transformer contains 6 decoder layers. After passing through the decoder layers, a linear layer is applied. Then Softmax converts the final values into probabilities. Decoder Output ↓ Linear Layer ↓ Softmax ↓ Probability of each token ↓ Most suitable next token For example: I am a ______ boy → 0.72 girl → 0.12 student → 0.08 doctor → 0.03 ... The model selects a token according to the generation strategy being used. Here is the complete flow of the original encoder-decoder Transformer: INPUT SENTENCE ↓ TOKENIZATION ↓ EMBEDDING ↓ POSITIONAL ENCODING ↓ ┌──────────────────────┐ │ ENCODER × 6 │ │ │ │ Multi-Head Attention │ │ ↓ │ │ Add & Norm │ │ ↓ │ │ FFN │ │ ↓ │ │ Add & Norm │ └──────────┬───────────┘ ↓ FINAL ENCODER OUTPUT │ │ ↓ TARGET → Embedding + Positional Encoding ↓ ┌──────────────────────┐ │ DECODER × 6 │ │ │ │ Masked Self-Attention│ │ ↓ │ │ Add & Norm │ │ ↓ │ │ Cross-Attention ←┘ │ ↓ │ Add & Norm │ ↓ │ FFN │ ↓ │ Add & Norm └──────────┬───────────┘ ↓ Linear Layer ↓ Softmax ↓ OUTPUT TOKEN ↓ NEXT TOKEN ... The Transformer can now be understood as a sequence of operations: Words ↓ Tokens ↓ Embeddings ↓ Positional Information ↓ Self-Attention ↓ Multi-Head Attention ↓ Feed-Forward Network ↓ Repeat through encoder layers ↓ Final Encoder Representation ↓ Decoder ↓ Masked Self-Attention ↓ Cross-Attention with Encoder Output ↓ Feed-Forward Network ↓ Repeat through decoder layers ↓ Linear + Softmax ↓ Output The most important idea is that attention allows the model to decide which other tokens are relevant when building the representation of a particular token. Transformers have become extremely important in modern AI. Some applications include: Machine translation Text generation Question answering Text summarization Sentiment analysis Code generation Image understanding Multimodal AI Models such as BERT, GPT and Vision Transformers are based on the Transformer idea, although their architectures can differ from the original encoder-decoder Transformer. Transformers changed the way we process sequential data by introducing attention as the central mechanism for understanding relationships between tokens. Instead of processing information strictly one token after another like traditional recurrent architectures, Transformers can use attention to determine which tokens are important to each other. The major components are: Tokenization ↓ Embeddings ↓ Positional Encoding ↓ Self-Attention ↓ Multi-Head Attention ↓ Feed-Forward Network ↓ Residual Connections + Layer Normalization ↓ Encoder / Decoder ↓ Cross-Attention ↓ Linear + Softmax ↓ Output Understanding these components makes it much easier to understand modern architectures such as BERT, GPT and other Transformer-based models. This is how Transformers work — from converting words into vectors to using attention to understand relationships and finally generating an output.