As a software engineer, my first experience with Large Language Models was the same as most developers — I called an API, sent a prompt, and received a response.
It was incredibly powerful, but it also felt like a black box. I knew how to use LLMs, but I didn’t understand how they actually worked.
Every time I pressed Enter, I found myself asking the same question:
What really happens between sending a prompt and seeing a response appear?
That question marked the beginning of my transition from building applications with AI to understanding how AI itself works.
Instead of stopping at API integrations, I started reading research papers, exploring the Transformer architecture, running open-source language models locally, and implementing many of the core concepts from scratch. I wanted to move beyond simply using LLMs and understand the mathematical and engineering ideas that make them work.
This article is a collection of what I learned during that journey. Rather than treating an LLM as a mysterious black box, we’ll open it up piece by piece — from neural networks and embeddings to attention, transformers, and production inference.
Whether you’re a software engineer beginning your AI journey or simply curious about what happens behind the scenes, my goal is to help you build an intuition for how modern language models actually work.
Before diving into individual concepts like tokenization, embeddings, and attention, let’s look at the entire journey of a prompt.
Imagine you’re asking ChatGPT a question:
“Explain quantum computing like I’m 10 years old.”
You hit Send, but the model doesn’t immediately “understand” your sentence like a human would. Instead, it processes your request through a series of mathematical steps.
Here’s the complete pipeline:
You type a prompt │ ▼1. Split the text into tokens │ ▼2. Convert each token into embeddings (vectors) │ ▼3. Process the vectors through transformer layers (using self-attention) │ ▼4. Predict the most likely next token │ ▼5. Append the predicted token and repeat │ ▼Final response appears one token at a time
For our example, the process looks something like this:
You type:"Explain quantum computing"
↓
Tokens:["Explain"] ["quantum"] ["computing"]
↓
Embeddings:[vector] [vector] [vector]
↓
Transformer thinks:"What should come next?"
↓
Predicts:"Quantum"
↓
Repeats...
"Quantum computing uses qubits..."
Notice something interesting: the model doesn’t generate the entire paragraph at once. It predicts one token at a time, adds it to the sentence, then uses the updated text to predict the next token. This loop continues until the response is complete.
Whether you’re chatting with ChatGPT, generating code, writing an email, or translating text, every response follows this same pipeline. The rest of this article explores each of these steps in detail.
Before we can understand LLMs, we need to understand how models learn in the first place.
A neural network is a mathematical system of connected layers. Each connection has a weight — a number that determines how much one neuron influences the next. When the model makes a prediction, it runs input data through these layers (called a forward pass), compares its output to the expected answer using a loss function, and adjusts its weights to reduce the error next time.
This cycle — forward pass, measure error, backpropagate, update weights — is the fundamental loop behind every deep learning model, from a single neuron to GPT-4.
I built a simple neural network to see this in action. Five training examples, 50 epochs:
class SimpleNN(nn.Module): def __init__(self): super().__init__() self.linear = nn.Linear(2, 1) def forward(self, x): return self.linear(x)
model = SimpleNN()loss_fn = nn.MSELoss()optimizer = optim.SGD(model.parameters(), lr=0.01)
for epoch in range(50): for x_data, y_true in training_data: prediction = model(torch.tensor([x_data])) loss = loss_fn(prediction, torch.tensor([[y_true]])) optimizer.zero_grad() loss.backward() optimizer.step()
Epoch 0 | Loss: 14.0335Epoch 10 | Loss: 0.0124Epoch 50 | Loss: 0.0052
Test on unseen [2, 3] → Predicted: 5.11 (expected: 5.0)
The loss dropped from 14 to near zero. And critically, the model could generalize — it correctly predicted an input it had never seen during training. This ability to generalize is what makes neural networks useful.
Computers don’t understand words. They understand numbers. So before any text enters a language model, it must be converted into numerical form. This is done by splitting the text into tokens — discrete units that the model can process.
GPT-2 uses Byte Pair Encoding (BPE), a subword tokenization algorithm. Common words stay whole. Rare or unknown words are broken into smaller pieces. This keeps the vocabulary manageable while still handling any input text.
For example:
"My firewall keeps blocking"
↓ BPE Tokenization
['My', 'Ġfirewall', 'Ġkeeps', 'Ġblocking']
The Ġ symbol represents a space. This is how the model knows these are separate words.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")text = "My firewall keeps blocking"tokens = tokenizer(text, return_tensors="pt")
print(tokenizer.convert_ids_to_tokens(tokens["input_ids"][0]))
Each token is then mapped to a numerical ID. The model never sees the word “firewall” — it sees a number like 12345. Everything downstream works with these numeric IDs.
Once text has been split into tokens, the next challenge is representing those tokens in a way a neural network can process.
A token ID is simply an integer. For example, the IDs assigned to “doctor” and “nurse” are just arbitrary numbers — they don’t contain any information about the meanings of those words.
To capture meaning, each token ID is converted into an embedding: a vector of learned numbers.
For example, a simplified embedding might look like this:
doctor → [0.82, -0.15, 0.43, ...]nurse → [0.79, -0.18, 0.40, ...]mountain → [-0.34, 0.91, -0.62, ...]
These vectors are greatly simplified. In GPT-2, every embedding contains 768 learned values. The individual numbers aren’t meaningful on their own — the model learns meaningful patterns from how entire vectors relate to one another.
One way to think about embeddings is as a map of language. Words that appear in similar contexts naturally end up close together, while unrelated concepts are placed farther apart.
Similar Meaning
doctor ●────────● nurse
teacher ●───────● student
● mountain
This organization allows the model to measure semantic similarity mathematically. For example, doctor and nurse occupy nearby regions because they frequently occur in medical contexts, whereas mountain is much farther away because it belongs to a different concept.
Embeddings also capture relationships between words. A famous example is:
King − Man + Woman ≈ Queen
Similarly,
Actor − Male + Female ≈ ActressPrince − Male + Female ≈ Princess
These relationships aren’t hardcoded. They emerge naturally during training as the model learns statistical patterns from billions of words.
Before a token reaches the transformer layers, GPT-2 converts it into a 768-dimensional embedding vector. From that point onward, the model no longer processes words as text — it processes these vectors, which become the foundation for the attention mechanism discussed in the next section.
Imagine you’re in a classroom and the teacher says, “It’s complicated.” How do you know what “It” refers to? You look at what was said before.
That’s exactly what self-attention does. Every word in a sentence looks at every other word and decides which ones matter most for understanding its own meaning.
The mechanism works through three vectors:
Each token asks its Query against every other token’s Key. The matches determine which Values get passed to the next layer.
I analyzed GPT-2’s attention on the sentence:
“The firewall blocked the traffic because it was suspicious”
When processing the word “it”, the model’s attention distribution was:
Token AttentionThe 0.6787Ġfirewall 0.0700Ġblocked 0.0568Ġbecause 0.0601
68% of attention went to “The”. The model also checked “firewall” and “blocked” to confirm context. This is how the model resolves pronouns and understands references.
with torch.no_grad(): outputs = model(**inputs)
attentions = outputs.attentionslast_layer_attn = attentions[-1][0].mean(dim=0)
for i, tok in enumerate(tokens): weight = last_layer_attn[target_idx, i].item() if weight > 0.01: print(f" {tok:15s} {weight:.4f}")
So far, we’ve discussed the ideas behind neural networks, embeddings, and transformers. But how do developers actually build and run these models?
That’s where PyTorch comes in.
PyTorch is an open-source deep learning framework that provides the building blocks for creating, training, and running neural networks. Instead of implementing complex mathematical operations from scratch, you use PyTorch to handle tensors, matrix multiplications, automatic differentiation, and GPU acceleration.
In fact, most modern open-source LLMs — including GPT-2, Llama, Mistral, and Gemma — are built using PyTorch.
Think of it like this:
If you want to understand or work with LLMs beyond calling an API, PyTorch is an essential tool.
A tensor is PyTorch’s fundamental data structure. You can think of it as a more powerful version of a NumPy array.
Everything inside an LLM is represented as tensors:
For example:
import torch
x = torch.tensor([1, 2, 3])
This creates a tensor containing three numbers. While this example is simple, the same type of tensor is used to represent thousands or even millions of values inside a language model.
Training a neural network means adjusting millions (or even billions) of weights to reduce prediction errors.
Calculating these updates manually would be impossible.
PyTorch solves this with Autograd, an automatic differentiation system. During training, it records every mathematical operation performed in the forward pass. After computing the loss, it automatically calculates how each weight contributed to the error by computing gradients.
Instead of writing calculus by hand, you simply call:
loss.backward()
PyTorch computes all the gradients needed for learning.
Every neural network in PyTorch is built by extending the nn.Module class.
Whether you’re creating:
they all inherit from nn.Module.
This provides a standard structure for defining layers, storing parameters, and running the forward pass.
A typical PyTorch training loop looks like this:
Input Data │ ▼Convert to Tensors │ ▼Model Forward Pass │ ▼Calculate Loss │ ▼Autograd Computes Gradients │ ▼Optimizer Updates Weights │ ▼Repeat
Although this workflow looks simple, it powers everything from small neural networks to today’s largest language models.
In my implementation, I trained a small neural network in about 20 lines of PyTorch code. The exact same workflow — creating tensors, performing a forward pass, calculating loss, computing gradients with Autograd, and updating parameters — is used when fine-tuning models like GPT-2 or Llama. The scale changes dramatically, but the underlying process remains the same.
The same Transformer architecture can produce completely different behavior depending on how it’s trained after initial pretraining.
Base model (GPT-2) — predicts the next token:
Prompt: "My firewall keeps blocking"→ "the Internet. It's a simple matter of setting up a firewall on your computer."
Instruction-tuned model (Qwen2.5–0.5B-Instruct) — fine-tuned to follow directions:
Question: "Why does a firewall sometimes block legitimate traffic?"→ "When a firewall blocks legitimate traffic, it could be due to several reasons: 1. Policy Violation: The firewall might have blocked legitimate traffic..."
and running these models takes just a few lines using Hugging Face:
from transformers import pipeline
We just adapt the model for our use case.
Training a neural network from scratch is expensive. It takes massive amounts of data, serious compute power, and a lot of time.
That’s where transfer learning comes in.
Instead of starting from zero, you take a model that’s already been trained on a broad task and adapt it for something more specific. You’re not building from scratch — you’re building on top of something that already works.
The easiest way to understand this is skill reuse. Imagine you already know how to ride a bicycle. Learning a motorcycle becomes much easier because you’re not starting from zero — you’re just adapting what you already know.
Transfer learning works the same way. A pretrained model has already learned general patterns in data. When you fine-tune it for your use case, it learns much faster with far less effort.
This is how most modern AI works today. Big companies train massive foundation models once, and then developers adapt them for specific tasks. That’s why you can build powerful AI applications without needing billions of data points or insane compute.
The context window is the model’s short-term memory. It defines how many tokens the model can “see” at once. Everything inside this window is available for generating the next token. Everything outside is invisible.
GPT-2’s context window is 1,024 tokens. Modern models vary:
When a conversation exceeds the context window, older messages are dropped. This is why long chat sessions eventually lose track of earlier topics. It’s not a bug — it’s a fundamental architectural constraint.
I tested how context size affects output quality:
6 tokens → Generic continuation15 tokens → Somewhat relevant22 tokens → Coherent scene continuation34 tokens → Contextually appropriate response
LLMs don’t deterministically pick the same next token every time. They calculate probabilities and then sample from those probabilities. Generation parameters control how that sampling works.
Temperature controls the randomness of token selection:
Same prompt, different temperatures:
Temperature 0.3: "the other side of your connection..."Temperature 0.7: "the new version of Firefox..."Temperature 1.2: "that process from the network..."Temperature 1.5: "any browser that will access the PGP signature..."
Instead of considering all possible tokens, top-p limits selection to the smallest set of tokens whose cumulative probability exceeds a threshold (e.g., 0.9). This prevents the model from considering very unlikely tokens.
Discourages the model from repeating the same words or phrases. A small increase (1.1–1.2) significantly improves output quality.
Because the model samples probabilistically. Setting temperature=0 and top_p=1 makes it deterministic. Any variation in these parameters changes the output — even with the exact same prompt.
This is one of the first things you notice when using AI seriously.
Sometimes the model gives you an answer that sounds completely confident — but turns out to be wrong. It might reference a research study that doesn’t exist, suggest an API that was never created, or present a made-up fact as if it’s common knowledge. This is called a hallucination.
Why does this happen?
Because at its core, a language model isn’t trying to tell the truth. It’s trying to generate the most probable next piece of text. It has learned patterns from massive amounts of data, and its job is to continue those patterns in a way that feels natural and coherent.
But it doesn’t verify whether what it’s saying is correct.
So if a false statement looks like something that should come next based on the patterns it learned, the model will generate it with full confidence. It’s not lying — it has no concept of truth. It’s just doing what it was trained to do.
This is why you can’t blindly trust LLM outputs, especially for factual answers, code, or important decisions. Real-world systems try to reduce hallucination by grounding the model in actual data — connecting it to trusted documents, using retrieval-augmented generation (RAG), or asking it to cite sources.
The model is incredibly good at sounding right. But it still needs you to check if it actually is right.
I built a single application that connects every concept in this article. You type a sentence, pick a focus word, and it shows the entire pipeline in action:
Input: "The firewall blocked the traffic because it was suspicious"Focus word: "it"
TOKENIZATION 0: The 1: Ġfirewall 2: Ġblocked 3: Ġthe 4: Ġtraffic 5: Ġbecause 6: Ġit 7: Ġwas 8: Ġsuspicious
ATTENTION from 'it' The 0.6787 ████████████████████ Ġfirewall 0.0700 ██ Ġblocked 0.0568 ██ Ġbecause 0.0601 ██
NEXT TOKEN PREDICTION the -94.332 a -96.112 not -96.234
Understanding how LLMs work under the hood is step one. The next steps go deeper into customization and optimization:
Fine-Tuning — Teaching a base model new skills using your own data. This is how you turn a general-purpose model into a domain-specific assistant.
LoRA (Low-Rank Adaptation) — A technique that fine-tunes models with minimal memory by only updating a small subset of parameters.
QLoRA — Combines LoRA with quantization, allowing you to fine-tune models on consumer GPUs.
RLHF (Reinforcement Learning from Human Feedback) — Aligning model outputs with human preferences through reward-based training.
RAG (Retrieval-Augmented Generation) — Connecting LLMs to external knowledge sources so they can answer questions based on actual documents rather than memorized patterns.
Each of these builds on the foundation we’ve covered here. Understanding the fundamentals means the advanced topics are just variations on familiar concepts.
The complete codebase for these experiments is available at:https://github.com/NehaKhann/ai-engineering-journey
Understanding Large Language Models: From Neural Networks to Production Inference was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.