Transformers Explained Visually The Transformer neural network architecture, introduced in the 2017 paper "Attention is All You Need," has become the go-to deep learning architecture for text-generative models including OpenAI's GPT, Meta's Llama, and Google's Gemini, and is also applied to audio generation, image recognition, protein structure prediction, and game playing. Text-generative Transformers operate on next-token prediction, with their core innovation being the self-attention mechanism that captures long-range dependencies; the Transformer Explainer tool is powered by OpenAI's GPT-2 (small) model with 124 million parameters. Every text-generative Transformer consists of three key components: embedding, the Transformer block (attention mechanism plus MLP layer), and output probabilities produced by final linear and softmax layers. What is a Transformer? Transformer is a neural network architecture that has fundamentally changed the approach to Artificial Intelligence. Transformer was first introduced in the seminal paper "Attention is All You Need" https://dl.acm.org/doi/10.5555/3295222.3295349 in 2017 and has since become the go-to architecture for deep learning models, powering text-generative models like OpenAI's GPT , Meta's Llama , and Google's Gemini . Beyond text, Transformer is also applied in audio generation https://huggingface.co/learn/audio-course/en/chapter3/introduction , image recognition https://huggingface.co/learn/computer-vision-course/unit3/vision-transformers/vision-transformers-for-image-classification , protein structure prediction https://elifesciences.org/articles/82819 , and even game playing https://www.deeplearning.ai/the-batch/reinforcement-learning-plus-transformers-equals-efficiency/ , demonstrating its versatility across numerous domains. Fundamentally, text-generative Transformer models operate on the principle of next-token prediction : given a text prompt from the user, what is the most probable next token a word or part of a word that will follow this input? The core innovation and power of Transformers lie in their use of self-attention mechanism, which allows them to process entire sequences and capture long-range dependencies more effectively than previous architectures. GPT-2 family of models are prominent examples of text-generative Transformers. Transformer Explainer is powered by the GPT-2 https://huggingface.co/openai-community/gpt2 small model which has 124 million parameters. While it is not the latest or most powerful Transformer model, it shares many of the same architectural components and principles found in the current state-of-the-art models making it an ideal starting point for understanding the basics. Transformer Architecture Every text-generative Transformer consists of these three key components : 1. Embedding : Text input is divided into smaller units called tokens, which can be words or subwords. These tokens are converted into numerical vectors called embeddings, which capture the semantic meaning of words. 2. Transformer Block is the fundamental building block of the model that processes and transforms the input data. Each block includes: - Attention Mechanism , the core component of the Transformer block. It allows tokens to communicate with other tokens, capturing contextual information and relationships between words. - MLP Multilayer Perceptron Layer , a feed-forward network that operates on each token independently. While the goal of the attention layer is to route information between tokens, the goal of the MLP is to refine each token's representation. 3. Output Probabilities : The final linear and softmax layers transform the processed embeddings into probabilities, enabling the model to make predictions about the next token in a sequence. Embedding Let's say you want to generate text using a Transformer model. You add the prompt like this one: “Data visualization empowers users to” . This input needs to be converted into a format that the model can understand and process. That is where embedding comes in: it transforms the text into a numerical representation that the model can work with. To convert a prompt into embedding, we need to 1 tokenize the input, 2 obtain token embeddings, 3 add positional information, and finally 4 add up token and position encodings to get the final embedding. Let’s see how each of these steps is done. Step 1: Tokenization Tokenization is the process of breaking down the input text into smaller, more manageable pieces called tokens. These tokens can be a word or a subword. The words "Data" and "visualization" correspond to unique tokens, while the word "empowers" is split into two tokens. The full vocabulary of tokens is decided before training the model: GPT-2's vocabulary has 50,257 unique tokens. Now that we split our input text into tokens with distinct IDs, we can obtain their vector representation from embeddings. Step 2. Token Embedding GPT-2 small represents each token in the vocabulary as a 768-dimensional vector; the dimension of the vector depends on the model. These embedding vectors are stored in a matrix of shape 50,257, 768 , containing approximately 39 million parameters This extensive matrix allows the model to assign semantic meaning to each token, in the sense that tokens with similar usage or meaning in language are placed close together in this high-dimensional space, while dissimilar tokens are farther apart. Step 3. Positional Encoding The Embedding layer also encodes information about each token's position in the input prompt. Different models use various methods for positional encoding. GPT-2 trains its own positional encoding matrix from scratch, integrating it directly into the training process. Step 4. Final Embedding Finally, we sum the token and positional encodings to get the final embedding representation. This combined representation captures both the semantic meaning of the tokens and their position in the input sequence. Transformer Block The core of the Transformer's processing lies in the Transformer block, which comprises multi-head self-attention and a Multi-Layer Perceptron layer. Most models consist of multiple such blocks that are stacked sequentially one after the other. The token representations evolve through layers, from the first block to the last one, allowing the model to build up an intricate understanding of each token. This layered approach leads to higher-order representations of the input. The GPT-2 small model we are examining consists of 12 such blocks. Multi-Head Self-Attention The self-attention mechanism enables the model to capture relationships among tokens in a sequence, so that each token’s representation is influenced by the others. Multiple attention heads allow the model to consider these relationships from different perspectives; for example, one head may capture short-range syntactic links while another tracks broader semantic context. In the following section, we will walk through how multi-head self-attention is computed step by step. Step 1: Query, Key, and Value Matrices Each token's embedding vector is transformed into three vectors: Query Q , Key K , and Value V . These vectors are derived by multiplying the input embedding matrix with learned weight matrices for Q, K, and V. Here's a web search analogy to help us build some intuition behind these matrices: - Query Q is the search text you type in the search engine bar. This is the token you want to "find more information about" . - Key K is the title of each web page in the search result window. It represents the possible tokens the query can attend to. - Value V is the actual content of web pages shown. Once we matched the appropriate search term Query with the relevant results Key , we want to get the content Value of the most relevant pages. By using these QKV values, the model can calculate attention scores, which determine how much focus each token should receive when generating predictions. Step 2: Multi-Head Splitting Query, key, and Value vectors are split into multiple heads—in GPT-2 small 's case, into 12 heads. Each head processes a segment of the embeddings independently, capturing different syntactic and semantic relationships. This design facilitates parallel learning of diverse linguistic features, enhancing the model's representational power. Step 3: Masked Self-Attention In each head, we perform masked self-attention calculations. This mechanism allows the model to generate sequences by focusing on relevant parts of the input while preventing access to future tokens. - Dot Product : The dot product of Query and Key matrices determines the attention score , producing a square matrix that reflects the relationship between all input tokens. - Scaling · Mask : The attention scores are scaled and a mask is applied to the upper triangle of the attention matrix to prevent the model from accessing future tokens, setting these values to negative infinity. The model needs to learn how to predict the next token without “peeking” into the future. - Softmax · Dropout : After masking and scaling, the attention scores are converted into probabilities by the softmax operation, then optionally regularized with dropout. Each row of the matrix sums to one and indicates the relevance of every other token to the left of it. Step 4: Output and Concatenation The model uses the masked self-attention scores and multiplies them with the Value matrix to get the final output of the self-attention mechanism. GPT-2 has 12 self-attention heads, each capturing different relationships between tokens. The outputs of these heads are concatenated and passed through a linear projection. MLP: Multi-Layer Perceptron After the multiple heads of self-attention capture the diverse relationships between the input tokens, the concatenated outputs are passed through the Multilayer Perceptron MLP layer to enhance the model's representational capacity. The MLP block consists of two linear transformations with a GELU