Most people use ChatGPT and Claude every day and have no idea how these systems are actually built. That is not a moral failure; the field is simply wrapped in strange language. Tokens, pretraining, fine-tuning, reward models, RLHF, alignment — it sounds like a wall of math, but the core pipeline is not impossible to understand.
You do not need to be a frontier-lab researcher to build a clear mental model. You just need one clean map. Here is the honest version: you are not going to train the next GPT or Claude from scratch on your laptop. Frontier models require massive datasets and millions in compute. The real goal is to understand the pipeline so well that you can build a tiny version yourself and stop treating AI like magic. Once you see the machine, you use it differently.
This guide is for AI builders, developers, technical founders, and curious power users who want to understand what’s actually happening under the hood of modern language models.
This is for you if:
✅ You want a simple mental model of the LLM pipeline
✅ You want to understand why models hallucinate or why some feel more helpful than others
✅ You want to build a small learning version instead of just reading theory
This is not a guide to competing with OpenAI or Anthropic. This is a guide to understanding the shape of the system.
By the end of this guide, you’ll understand:
✅ How raw text becomes tokens — and why models don’t “read” like humans
✅ Why pretraining is next-token prediction — the absurdly simple objective behind LLMs
✅ Why base models hallucinate — and why fluency ≠ truth
✅ How SFT turns a raw model into an assistant — teaching taste, not just facts
✅ What reward models, DPO, and RLHF actually do — the secret sauce
✅ Why alignment improves behavior — but adds tradeoffs
Before diving deep, here is the bird’s-eye view of how a raw pile of text becomes a helpful assistant:
Now let us walk through it.
Before there is a model, there is text — a lot of it. Beginners usually misunderstand this stage. They think the hard part is collecting more data, when the harder part is actually deciding what data deserves to stay. Raw data is messy (spam, duplicates, toxic content, boilerplate). The model becomes the statistical shape of what it sees. This is why Data Contamination (when test data leaks into training) and Deduplication are critical engineering tasks.
Models don’t see words; they see Tokens — chunks of text converted into integer IDs. This explains the mystery of why a model can write brilliant code but struggles to count letters in a word. It’s not a spelling machine; it’s a token sequencer.
🔧 What to Build Yourself: Take a small dataset, clean it, and run it through OpenAI’s **tiktoken **or a Hugging Face tokenizer. Watch how normal sentences split into integers.
The objective here is almost absurdly simple: predict the next token. The model sees a sequence, guesses the next token, and its parameters are nudged slightly if it’s wrong. Repeat billions of times.
To predict the next token accurately, the model implicitly learns grammar, facts, Python syntax, and cause-and-effect. The output is a raw “Base Model.” It has language ability but no “assistant” behavior. It doesn’t answer questions; it just completes text plausibly. This is exactly why hallucinations happen: the engine is optimized for plausibility, not truth. Never treat fluency as proof of accuracy.
🔧 What to Build Yourself: Train a tiny character-level language model using Andrej Karpathy’s nanoGPT in a Colab notebook. Feel the core loop: Input → Predict → Compare → Adjust → Repeat.
Now you have a base model, but you want an assistant. The shift is simple: show the model examples of the behavior you want (Question → Good Answer). The objective is still token prediction, but the data is now highly curated.
SFT requires much less data than pretraining, but its quality dictates the model’s “personality.” If your examples are vague, the model becomes vague. Fine-tuning teaches taste, not just information. However, there is a risk: Catastrophic Forgetting. If you fine-tune too aggressively on a narrow task, the model forgets its broader base skills.
🔧 What to Build Yourself: Build a tiny 100-example instruction dataset. Fine-tune a small open model (like Llama 3 8B) using **Unsloth **or **Axolotl **via QLoRA on a single GPU.
Two answers can be factually correct, but one is clearer, safer, and admits uncertainty. You can’t write a hardcoded rule for this, so you collect human preference data by ranking multiple model outputs.
In classic RLHF, this data trains a Reward Model — an automated judge that scores answers at scale. But if the reward model is flawed, the main model learns loopholes to get high scores while giving terrible answers. This is called Reward Hacking. Modern alternatives like Direct Preference Optimization (DPO) skip the separate reward model and train directly on the rankings.
🔧 What to Build Yourself: Take 50 prompts, generate 4 answers for each, and rank them. Look into DPO tutorials to see how to train on these pairs.
This is the final feedback loop: Generate → Score → Improve. Using the preference signal, the model learns nuance, safety, refusals, and its distinct tone.
However, there is a cost: the Alignment Tax. Making a model safer and more cautious can sometimes reduce its raw capabilities on complex, unstructured tasks. This stage is what makes two models with similar base capabilities feel completely different to the user.
🔧 What to Build Yourself: Follow a basic DPO tutorial using your Stage 4 dataset to see how feedback visibly shifts model behavior.
Once you stop seeing the model as a magic brain and start seeing it as a trained system, your relationship with AI changes completely:
Stage 1: Data & Tokenization
- OpenAI tiktoken: [github.com/openai/tiktoken](https://github.com/openai/tiktoken)
- Hugging Face Tokenizers: [huggingface.co/docs/tokenizers](https://huggingface.co/docs/tokenizers)
Stage 2: Pretraining
- Andrej Karpathy’s nanoGPT: [github.com/karpathy/nanoGPT](https://github.com/karpathy/nanoGPT)
- His YouTube tutorial: [www.youtube.com/watch?v=kCc8FmEb1nY](https://www.youtube.com/watch?v=kCc8FmEb1nY)
**Stage 3: Supervised Fine-Tuning**
- Unsloth (fast fine-tuning): [github.com/unslothai/unsloth](https://github.com/unslothai/unsloth)
- Axolotl: [github.com/axolotl-ai-cloud/axolotl](https://github.com/axolotl-ai-cloud/axolotl)
Stage 4–5: Preference & Alignment
- Hugging Face TRL (DPO tutorials): [huggingface.co/docs/trl](https://huggingface.co/docs/trl)
- DPO paper: [arxiv.org/abs/2305.18290](https://arxiv.org/abs/2305.18290)
Start with Stage 1. Build the smallest version you can. Then move through the pipeline one stage at a time.
— -
Enjoyed this guide?
Follow me for more practical AI engineering content.
If you build a tiny LLM using this pipeline, share your results in the comments — I’d love to see what you create. How to Build Your Own Tiny LLM From Scratch was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.