{"slug": "from-neural-networks-to-llms-the-mental-model-i-was-missing", "title": "From Neural Networks to LLMs: The Mental Model I Was Missing", "summary": "A developer explains the mental model connecting neural networks, deep learning, Transformers, and attention to understand how LLMs work. The post traces the evolution from basic neural networks to the Transformer architecture introduced in 'Attention Is All You Need', highlighting self-attention as a core mechanism for capturing context in language.", "body_md": "Before jumping into APIs, RAG, agents, and AI applications, I wanted to understand what actually happens inside an LLM.\n\nI kept coming across terms like **neural networks, deep learning, Transformers, attention, tokens, embeddings, BERT, GPT, and causal language modeling** — but they all felt like disconnected pieces.\n\nI could understand each concept individually, but I didn't have a clear picture of how they all connected.\n\nSo I decided to step back and build the mental model from the ground up.\n\nThis article is my attempt to connect those pieces.\n\nBefore understanding LLMs, it helps to understand where they come from.\n\nA neural network is a machine learning model that learns patterns from data.\n\nFor example, suppose we want to recognize handwritten digits.\n\nWe could give a neural network thousands of images of handwritten digits:\n\n```\nImages of handwritten digits\n            ↓\n      Neural Network\n            ↓\n     Learned patterns\n            ↓\n        Prediction\n```\n\nInstead of manually programming rules such as:\n\n\"If the image has this curve and this line, it must be a 3.\"\n\nwe allow the neural network to learn those patterns from examples.\n\nThe things the network learns are stored in its **parameters**, primarily weights and biases.\n\nAs we increase the number of layers in a neural network, we enter the world of **deep learning**.\n\n```\nMachine Learning\n       ↓\nNeural Networks\n       ↓\nDeep Learning\n```\n\nDifferent neural network architectures became useful for different types of problems.\n\nFor example:\n\nBut language has a special challenge.\n\nConsider:\n\n\"The animal didn't cross the road because it was tired.\"\n\nTo understand what **\"it\"** refers to, we need to understand its relationship with the other words in the sentence.\n\nThis is where things like **attention** become important.\n\nAnd this eventually leads us to Transformers.\n\nBefore Transformers, RNNs and LSTMs were commonly used for language-related tasks.\n\nThey processed sequences step by step.\n\nFor example:\n\n```\nI → love → machine → learning\n```\n\nThis sequential processing created some challenges.\n\nOne major problem was that it made training difficult to parallelize efficiently.\n\nIt could also become difficult to capture relationships between tokens that were far apart in a long sequence.\n\nThen, in 2017, researchers introduced the **Transformer architecture** in the paper:\n\n\"Attention Is All You Need\"\n\nThe original Transformer was designed for **machine translation**.\n\nThe basic architecture looked like this:\n\n```\nEnglish sentence\n       ↓\n    Encoder\n       ↓\nContextual representation\n       ↓\n    Decoder\n       ↓\nFrench sentence\n```\n\nFor example:\n\n```\n\"I love cats\"\n      ↓\n   Transformer\n      ↓\n\"J'aime les chats\"\n```\n\nThe key idea that made Transformers different was **attention**.\n\nInstead of processing a sequence strictly one step at a time, the Transformer could use attention to determine relationships between different tokens.\n\nThis made it much better suited to capturing context and also allowed much more parallel computation during training.\n\nAt a high level, attention answers:\n\n\"When I'm processing this token, which other tokens should I pay attention to?\"\n\nConsider:\n\n\"The animal didn't cross the road because it was tired.\"\n\nWhen processing **\"it\"**, the model needs to determine which other tokens are relevant.\n\nConceptually:\n\n```\nThe animal didn't cross the road because it was tired.\n                                      ↑\n                                     \"it\"\n```\n\nThe model can assign different levels of importance to different tokens.\n\nThis is called **self-attention**, because tokens in the sequence attend to other tokens in the same sequence.\n\nI initially thought attention was something separate from the Transformer.\n\nIt isn't.\n\nAttention is one of the core mechanisms inside a Transformer.\n\nFor now, the mental model I use is:\n\n```\nCurrent token\n     ↓\nLook at other relevant tokens\n     ↓\nCombine useful information\n     ↓\nCreate a contextual representation\n```\n\nThe mathematical details — Query, Key, Value, attention scores, and matrix multiplication — deserve their own article.\n\nThe original Transformer architecture contained two major components:\n\n```\nInput\n  ↓\nEncoder\n  ↓\nRepresentation\n  ↓\nDecoder\n  ↓\nOutput\n```\n\nA useful beginner mental model is:\n\nEncoder → processes and represents the input\n\nDecoder → generates the output\n\nFor translation:\n\n```\nEnglish\n   ↓\nEncoder\n   ↓\nRepresentation\n   ↓\nDecoder\n   ↓\nFrench\n```\n\nThis architecture was designed around a very natural problem:\n\nTake one sequence and transform it into another sequence.\n\nFor example:\n\nBut later, researchers realized that we don't always need both parts.\n\nThis led to different Transformer architectures.\n\nThis was one of the things I initially found confusing.\n\nIf BERT, GPT, T5, and BART are all based on Transformers, why are they different?\n\nThe answer is:\n\nTransformer is an architecture, not one specific model.\n\nDifferent models can use different parts of the Transformer architecture.\n\nA simple mental model is:\n\n```\n                         Transformer\n                              │\n              ┌───────────────┼───────────────┐\n              ↓               ↓               ↓\n        Encoder-only     Decoder-only    Encoder-decoder\n              ↓               ↓               ↓\n            BERT              GPT          T5 / BART\n```\n\nLet's look at each one.\n\nAn encoder-only model uses the encoder part of the Transformer.\n\nThe encoder processes the input and creates contextual representations.\n\nThe most famous example is **BERT**.\n\nBERT was trained using **Masked Language Modeling**.\n\nFor example:\n\n```\nThe cat is [MASK] on the mat.\n```\n\nThe model tries to predict the missing token.\n\nBecause the encoder can use context from both sides of the masked token, it can build a bidirectional representation.\n\nThis makes encoder-only models useful for tasks such as:\n\nThe simple mental model:\n\nEncoder-only → understand/represent text\n\nA decoder-only model uses the decoder part of the Transformer.\n\nThe most famous example is **GPT**.\n\nGPT stands for:\n\nGenerative Pre-trained Transformer\n\nGPT is trained primarily using **causal language modeling**.\n\nIts objective is:\n\nPredict the next token using the previous tokens.\n\nFor example:\n\n```\nThe cat is\n       ↓\n    sleeping\n```\n\nThen:\n\n```\nThe cat is sleeping\n                  ↓\n              next token\n```\n\nThe model keeps generating one token at a time.\n\nThis is why GPT is naturally suited to text generation.\n\nThe simple mental model:\n\nDecoder-only → generate text\n\nEncoder-decoder models use both parts.\n\nThe encoder processes the input.\n\nThe decoder generates the output.\n\nFor example, in summarization:\n\n```\nArticle\n   ↓\nEncoder\n   ↓\nRepresentation\n   ↓\nDecoder\n   ↓\nSummary\n```\n\nOr translation:\n\n```\nEnglish\n   ↓\nEncoder\n   ↓\nRepresentation\n   ↓\nDecoder\n   ↓\nFrench\n```\n\nExamples include **T5** and **BART**.\n\nThe simple mental model:\n\nEncoder-decoder → transform one sequence into another\n\nThis is the simplest table I use to remember them:\n\n| Model | Architecture | Simple mental model |\n|---|---|---|\n| BERT | Encoder-only | Understand / represent |\n| GPT | Decoder-only | Generate |\n| T5 | Encoder-decoder | Input → Output |\n| BART | Encoder-decoder | Understand → Generate |\n\nThe important thing to remember is:\n\nThey are different ways of using the Transformer architecture.\n\nNow we can finally talk about **Large Language Models**.\n\nLLM stands for:\n\nLarge Language Model\n\nLet's break down the name.\n\n\"Large\" generally refers to the enormous number of learned parameters.\n\nParameters are numerical values learned during training, primarily:\n\nModern language models can have billions of parameters.\n\nThe model is trained on large amounts of language data and learns patterns and relationships in that data.\n\nIt is ultimately a neural network that has learned these patterns through its parameters.\n\nSo:\n\n```\nLLM\n│\n├── Large\n│     └── Many learned parameters\n│\n├── Language\n│     └── Learns patterns from language\n│\n└── Model\n      └── Neural network\n```\n\nGPT-style models are generally **large decoder-only Transformer models**.\n\nThey are trained primarily using **causal language modeling**.\n\nThe basic objective is surprisingly simple:\n\nPredict the next token based on the previous context.\n\nSuppose the training data contains:\n\n```\nThe sky is blue.\n```\n\nThe model learns to predict:\n\n```\nThe\n ↓\nsky\n```\n\nThen:\n\n```\nThe sky\n ↓\nis\n```\n\nThen:\n\n```\nThe sky is\n ↓\nblue\n```\n\nThis happens across enormous amounts of training data.\n\nDuring training:\n\n```\nInput context\n     ↓\nPredict next token\n     ↓\nCompare with actual token\n     ↓\nCalculate error\n     ↓\nUpdate parameters\n     ↓\nRepeat\n```\n\nOver time, the model becomes better at predicting the next token.\n\nThis was one of the most interesting things for me to understand.\n\nIf GPT is fundamentally trained to predict the next token, how can it:\n\nThe answer is largely **context**.\n\nThe model doesn't necessarily need a separate mechanism for every task.\n\nThe prompt provides context about what kind of continuation is expected.\n\nFor example:\n\n```\nTranslate to French:\n\nI love cats.\n```\n\nThe context tells the model that the expected continuation is a translation.\n\nOr:\n\n```\nSummarize this article:\n\n[article]\n```\n\nNow the expected continuation is a summary.\n\nOr:\n\n```\nWrite a Python function that sorts a list.\n```\n\nNow the expected continuation is code.\n\nThe underlying mechanism is still:\n\n```\nContext\n   ↓\nPredict next token\n   ↓\nAdd token to context\n   ↓\nPredict next token\n   ↓\nAdd token to context\n   ↓\nRepeat\n```\n\nThis is one of the most important mental models I have taken away:\n\nA relatively simple training objective — next-token prediction — can result in a model capable of many different language tasks.\n\nAt this point, another question naturally appears:\n\nIf the model predicts tokens, does it actually process words directly?\n\nNo.\n\nThe model needs to convert our text into numerical representations that a neural network can process.\n\nThis starts with **tokenization**.\n\nFor example:\n\n```\n\"I love programming\"\n```\n\nmight become something conceptually like:\n\n```\n[\"I\", \" love\", \" programming\"]\n```\n\nThe exact tokens depend on the tokenizer.\n\nA token can be:\n\nSo:\n\nA token is not necessarily a word.\n\nThe token is then mapped to a numerical ID:\n\n```\nText\n ↓\nTokens\n ↓\nToken IDs\n```\n\nThe Transformer doesn't directly work with the token ID as a meaningful representation.\n\nThe token ID is essentially an index.\n\nThe model needs a richer numerical representation.\n\nThis is where **embeddings** come in.\n\nConceptually:\n\n```\n\"cat\"\n  ↓\nToken\n  ↓\nToken ID\n  ↓\nEmbedding\n  ↓\n[0.21, -0.42, 0.17, ...]\n```\n\nAn embedding is a vector containing many numerical values.\n\nFor example, if an embedding has 4,096 dimensions:\n\n```\n[0.21, -0.42, 0.17, ..., 0.31]\n```\n\nit simply means that the vector contains **4,096 numbers**.\n\nThese numbers allow the Transformer to perform mathematical operations on the representation of the token.\n\nAt this point, our mental model becomes:\n\n```\nText\n ↓\nTokenization\n ↓\nTokens\n ↓\nToken IDs\n ↓\nEmbeddings\n ↓\nTransformer\n```\n\nI'll go much deeper into tokenization and embeddings in the next article.\n\nAfter connecting all these concepts, this is the mental map I currently have:\n\n```\n                    Neural Networks\n                           ↓\n                     Deep Learning\n                           ↓\n                     Sequence Models\n                           ↓\n                      Transformers\n                           ↓\n                        Attention\n                           ↓\n              Transformer Architectures\n                           ↓\n          ┌────────────────┼────────────────┐\n          ↓                ↓                ↓\n     Encoder-only     Decoder-only    Encoder-decoder\n          ↓                ↓                ↓\n        BERT              GPT           T5 / BART\n                           ↓\n                          LLMs\n```\n\nAnd if I zoom into a GPT-style LLM:\n\n```\nUser Prompt\n     ↓\nTokenization\n     ↓\nToken IDs\n     ↓\nEmbeddings\n     ↓\nTransformer Blocks\n     ↓\nAttention\n     ↓\nContextual Representation\n     ↓\nNext-token prediction\n     ↓\nGenerated token\n     ↓\nRepeat\n```\n\nThis finally gave me the map I was missing.\n\nI don't need to know every mathematical detail yet.\n\nI first need to know **where each concept belongs**.\n\nNow comes the question I'm most interested in:\n\nWhat actually happens inside an LLM when I type a prompt?\n\nSuppose I ask:\n\n```\nWhat is the capital of India?\n```\n\nAt a very high level, the process looks something like:\n\n```\n\"What is the capital of India?\"\n             ↓\n        Tokenization\n             ↓\n         Token IDs\n             ↓\n         Embeddings\n             ↓\n   Positional Information\n             ↓\n     Transformer Blocks\n             ↓\n         Attention\n             ↓\n Contextual Representation\n             ↓\n        Output Scores\n             ↓\n          Softmax\n             ↓\n Probability Distribution\n             ↓\n        Next Token\n             ↓\n           Repeat\n```\n\nAnd this is where I want to go next.\n\nNow that I have the high-level map, I want to open up the black box.\n\nIn the next article, I'll start from the beginning of the inference process:\n\n```\nPrompt\n  ↓\nTokenization\n  ↓\nToken IDs\n  ↓\nEmbeddings\n  ↓\nPositional information\n  ↓\nTransformer\n  ↓\nAttention\n  ↓\nHidden states\n  ↓\nUnembedding\n  ↓\nLogits\n  ↓\nSoftmax\n  ↓\nProbability distribution\n  ↓\nNext token\n```\n\nI'll break down each step and answer questions like:\n\nThe goal is not to start with mathematics.\n\nThe goal is to first build an intuitive mental model and then introduce the mathematics once the pieces make sense.\n\nIf I had to summarize everything I've learned so far in one sentence:\n\nAn LLM like GPT is a large neural network built using the Transformer architecture, trained on language to predict the next token, and capable of many language tasks because of the patterns and representations it learns from enormous amounts of data.\n\nThe biggest thing that changed for me was realizing that all these terms aren't isolated concepts.\n\nThey fit together:\n\n```\nNeural Networks\n      ↓\nDeep Learning\n      ↓\nTransformers\n      ↓\nAttention\n      ↓\nEncoder / Decoder Architectures\n      ↓\nGPT\n      ↓\nLLMs\n      ↓\nTokens\n      ↓\nEmbeddings\n      ↓\nTransformer Processing\n      ↓\nNext-token Prediction\n      ↓\nGenerated Text\n```\n\nNow that I have the map, I'm ready to understand what's actually happening **inside the box**.", "url": "https://wpnews.pro/news/from-neural-networks-to-llms-the-mental-model-i-was-missing", "canonical_source": "https://dev.to/priyankaa/from-neural-networks-to-llms-a-developers-mental-model-55a6", "published_at": "2026-08-16 11:30:08+00:00", "updated_at": "2026-08-16 11:42:09.003932+00:00", "lang": "en", "topics": ["machine-learning", "neural-networks", "large-language-models", "natural-language-processing"], "entities": ["Transformer", "BERT", "GPT", "Attention Is All You Need"], "alternates": {"html": "https://wpnews.pro/news/from-neural-networks-to-llms-the-mental-model-i-was-missing", "markdown": "https://wpnews.pro/news/from-neural-networks-to-llms-the-mental-model-i-was-missing.md", "text": "https://wpnews.pro/news/from-neural-networks-to-llms-the-mental-model-i-was-missing.txt", "jsonld": "https://wpnews.pro/news/from-neural-networks-to-llms-the-mental-model-i-was-missing.jsonld"}}