In my previous article, we opened the hood of a transformer and traced what happens between pressing Enter and seeing a response appear. We explored neural networks, tokenization, attention, embeddings, and the generation pipeline.
But after all that, a natural question emerged:
I can prompt the model — but what if it doesn’t know my domain? What if it keeps generating nonsense instead of giving me a real answer?
That’s where ** fine-tuning** comes in. If Week 1 was about understanding how LLMs work, Week 2 is about making them work
Following along
Each day's code lives in its own folder in the repo, so you can jump to any part independently:
ai-engineering-journey/└── weeks/ └── week-02-fine-tuning-fundamentals/ ├── assets/ ├── day-01-prompt-engineering/ ├── day-02-instruction-tuning/ ├── day-03-dataset-preparation/ ├── day-04-supervised-fine-tuning/ ├── day-05-model-evaluation/ ├── README.md └── requirements.txt
All code is available at my repository:[github.com/NehaKhann/ai-engineering-journey]
Let me show you the exact moment I realized prompting has limits.
I loaded GPT-2 — a base model — and asked it a simple question:
from transformers import pipeline
base = pipeline("text-generation", model="gpt2")print(base("Explain what is a firewall in simple words.")[0]["generated_text"])
Here’s what GPT-2 returned:
Explain what is a firewall in simple words. The first thing you need to know is that the firewall is a very important part of a network. It’s a device that controls the…
This looks reasonable at first glance. But try again:
Explain what is a firewall in simple words. “I don’t know,” said Alice. “That’s the problem,” said the Queen. “You don’t know anything.”
Same prompt. Completely different result — and the second time, it didn’t even attempt an answer.
Here’s why: GPT-2 is a base model — it was trained purely to predict “what word comes next” across a massive pile of internet text. It has no concept that it’s being asked something. It just keeps typing whatever statistically tends to follow, whether that’s a technical explanation or a random line from Alice in Wonderland. Think of it as someone who’s read millions of books but was never taught how to have a conversation — they’ll keep talking, just not necessarily to you.
An instruction-tuned model like Qwen2.5–0.5B-Instruct behaves differently, because it went through an extra round of training specifically on “when someone asks something, answer it”:
instruct = pipeline("text-generation", model="Qwen/Qwen2.5-0.5B-Instruct")msg = [{"role": "user", "content": "Explain what is a firewall in simple words."}]print(instruct(msg, max_new_tokens=80)[0]["generated_text"][-1]["content"])
A firewall is like a security guard for your computer or network. It checks all the data trying to come in or go out, and only allows the safe stuff through — blocking anything that looks suspicious or harmful.
Same question. This time, an actual answer — because the model recognizes the pattern: user asks, assistant answers.
Each of these techniques works the same way under the hood: they change what’s sitting in front of the model right before it predicts the next word. Nothing about the model itself changes — only what you feed it.
Asking directly, cold, no examples. Like walking up to a stranger and asking a question, trusting they’ll figure out what you want.
msg = [{"role": "user", "content": "What is a firewall?"}]out = chat(msg, max_new_tokens=80, temperature=0.7)
Showing 2–3 examples of the pattern you want before your real question. Like showing someone two solved math problems before handing them a third, unsolved one — they don’t need new math knowledge, just the pattern.
msg = [ {"role": "user", "content": "What is an IP address?"}, {"role": "assistant", "content": "An IP address is a unique numerical label assigned to each device on a network..."}, {"role": "user", "content": "What is DNS?"}, {"role": "assistant", "content": "DNS stands for Domain Name System. It translates domain names into IP addresses..."}, {"role": "user", "content": "What is a firewall?"}]out = chat(msg, max_new_tokens=80, temperature=0.7)
The real question is the last message, preceded by two full example exchanges — the model reads all of it as one continuous pattern and simply continues it.
Asking the model to “think step by step” instead of jumping straight to an answer. Like following furniture assembly instructions one step at a time instead of guessing how the final product goes together — each step sets up the next.
msg = [{"role": "user", "content": "Explain what a firewall is. Think step by step: " "first what it protects, then how it filters traffic, then a simple analogy."}]out = chat(msg, max_new_tokens=120, temperature=0.7)
Assigning a persona via a system message. Same model, same knowledge, but the persona shapes vocabulary and depth — like the same person explaining their job differently to a 5-year-old vs. a job interviewer, same facts, totally different words.
msg = [ {"role": "system", "content": "You are a cybersecurity professor with 15 years of experience. " "You explain concepts clearly with real-world analogies for beginners."}, {"role": "user", "content": "What is a firewall?"}]out = chat(msg, max_new_tokens=100, temperature=0.7)
Unlike the other three, this instruction sticks for the whole conversation — not just one reply.
Temperature controls how strictly the model follows the most probable path. Low temperature (0.3) means deterministic, factual output. High temperature (1.2) means creative, varied output — and this applies across all four techniques above; it’s how confidently the model commits to its top choice at every step.
temps = [("Low (0.2)", 0.2), ("Balanced (0.7)", 0.7), ("Creative (1.2)", 1.2)]for label, temp in temps: msg = [{"role": "user", "content": "Give one benefit of using a firewall. Keep it to one sentence."}] out = chat(msg, max_new_tokens=40, temperature=temp)
The prompt stays identical across all three runs — only temperature changes, which isolates its effect:
Picture it as a single dial. On the left, the model plays it safe — good for facts and code. On the right, it takes more risks with word choice — good for brainstorming, but more likely to wander off-topic:
As temperature climbs, the model shifts from being an accurate reporter to being a more inventive writer — neither is “better,” they’re suited to different jobs.
All the prompting techniques we’ve explored have one thing in common: the model never changes. You’re only changing the instructions you give it, while its learned weights remain exactly the same.
Fine-tuning is different. Instead of changing the input, you train the model on your own examples, updating its weights so the new behavior becomes part of the model itself.
Prompting:Your Prompt + Frozen Model → OutputFine-Tuning:Training Examples + Model → Updated Model
With prompting, the model only follows your instructions for the current conversation. Start a new chat, and those instructions are gone.
With fine-tuning, the model learns from hundreds or thousands of training examples. Using gradient descent, its weights are updated so it consistently behaves the new way — even in future conversations.
Imagine a new employee.
Once trained, they no longer need the instruction sheet.
In short: Prompting guides the model’s existing knowledge, while fine-tuning teaches it new behavior.
For example, GPT-2 is a base language model trained mainly for next-token prediction. You can prompt it in different ways, but it won’t naturally behave like a helpful chatbot. Instruction-tuned models such as Qwen2.5-Instruct achieve that behavior because they were fine-tuned on instruction-following examples after pretraining.
Here’s something that tripped me up early on. You can’t just feed raw text to an instruction-tuned model and expect it to understand who’s speaking.
If you pass this to a model:
What is a firewall?
The model has no idea whether this is:
Remember, a language model only predicts the next token. It doesn’t inherently understand conversation roles like “user” and “assistant.”
Chat templates solve this by wrapping each message with special tokens that label who’s speaking. Think of them like stage directions in a script:
<|im_start|>userWhat is a firewall?<|im_end|>
Now the model sees: “The text between <|im_start|>user and <|im_end|> is something the user said."
Then when it generates a response:
<|im_start|>assistantA firewall monitors network traffic...<|im_end|>
The model knows: “Now I need to speak as the assistant because that’s what comes next.”
Every instruction-tuned model has its own template defining these markers. Qwen2.5 uses <|im_start|> and <|im_end|>. Other models use different tokens, but the concept is the same — special boundaries that turn a plain text string into a structured conversation.
Instead of manually inserting these tokens, you use apply_chat_template():
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
messages = [ {"role": "system", "content": "You are a cybersecurity expert."}, {"role": "user", "content": "What is a firewall?"}, {"role": "assistant", "content": "A firewall monitors network traffic and blocks unauthorized access."},]
text = tokenizer.apply_chat_template(messages, tokenize=False)print(text)
This takes your structured list of messages and inserts the correct special tokens automatically. The output looks like:
<|im_start|>systemYou are a cybersecurity expert.<|im_end|><|im_start|>userWhat is a firewall?<|im_end|><|im_start|>assistantA firewall monitors network traffic and blocks unauthorized access.<|im_end|>
The model reads this as one continuous string, but the markers tell it the structure.
Let me make this real. Suppose you come to the model and say:
System message:“You are a cybersecurity expert.”Your question:“Explain firewall to me as a beginner.”
You want the model to reply with something like:
“A firewall is like a security guard for your computer. It checks all incoming and outgoing traffic and blocks anything suspicious.”
Now imagine we’re training the model. We have hundreds of examples where we already know what the correct answer should be. One training example looks like this — the full conversation:
<|im_start|>systemYou are a cybersecurity expert.<|im_end|><|im_start|>userExplain firewall to me as a beginner.<|im_end|><|im_start|>assistantA firewall is like a security guard for your computer. It checks all incoming and outgoing traffic and blocks anything suspicious.<|im_end|>
During training, the model reads this entire conversation. It tries to predict every single token — the system message, your question, and the assistant’s response.
But here’s the thing: we don’t care if the model predicts the system message or your question correctly. Those are inputs we give it. We only care if it learns to generate the right answer.
[System: "You are a cybersecurity expert."] → ✗ ignored for training[User Neha: "Explain firewall to me..."] → ✗ ignored for training[Assistant: "A firewall is like a security..."] → ✓ model learns from this
The three roles — system, user, and assistant — are just labels that tell the template how to format the conversation:
The model reads all three for context but only updates its weights based on the assistant’s part.
This process is called loss masking.
The system and user messages are still provided as context, helping the model understand the conversation. But when the model makes mistakes during training, only errors in the assistant’s response are used to calculate the loss and update the weights.
Without loss masking, the model would waste training effort trying to “predict” the prompts you already gave it. Instead, it focuses on learning the behavior you actually want: generating high-quality responses.
Think of it this way:During an exam, the questions are already printed on the paper. You’re graded only on your answers — not on whether you can rewrite the questions correctly. Loss masking works the same way. The model sees the entire conversation, but it’s trained only on the assistant’s answers.
github.com/NehaKhann/ai-engineering-journey
dataset = [ {"instruction": "What is a firewall?", "response": "A firewall monitors and controls incoming and outgoing network traffic based on security rules."}, {"instruction": "Explain DNS.", "response": "DNS, or Domain Name System, translates human-readable domain names into IP addresses."}, {"instruction": "What is encryption?", "response": "Encryption converts plaintext data into ciphertext using an algorithm and a key."}, {"instruction": "Define phishing.", "response": "Phishing is a cyber attack where attackers impersonate legitimate entities to steal information."}, {"instruction": "What is two-factor authentication?", "response": "2FA adds an extra security layer by requiring two verification methods before granting access."},]print(f"Dataset: {len(dataset)} instruction-response pairs\n")token_counts = []for i, item in enumerate(dataset, 1): msg = [ {"role": "system", "content": "You are a cybersecurity expert."}, {"role": "user", "content": item["instruction"]}, {"role": "assistant", "content": item["response"]} ] formatted = tokenizer.apply_chat_template(msg, tokenize=False) tokens = tokenizer.encode(formatted) token_counts.append(len(tokens)) print(f" Ex {i}: {len(tokens):3d} tokens — {item['instruction']}")
Each row shows one example from your dataset. The token count is how many pieces the model splits that example into after applying the chat template.
{"instruction": "What is a firewall?", "response": "A firewall monitors network traffic."}
Without template — the model sees this as raw text to memorize:
What is a firewall? A firewall monitors network traffic.
It learns to predict BOTH the question and the answer equally. Wasted effort.
With template — the model sees:
<|im_start|>userWhat is a firewall?<|im_end|><|im_start|>assistantA firewall monitors network traffic.<|im_end|>
Those 41 tokens include the markers. Now the model knows:
Benefit: The model focuses 100% of its training on learning how to answer, not on memorizing questions. That's why after training, when you ask "What is a firewall?", it answers instead of continuing with random text.
The five examples above were enough to show how chat templates and loss masking work. But five examples won’t teach a model anything — real fine-tuning needs dozens, hundreds, or thousands of examples. And the moment you’re dealing with more than a handful, a new problem shows up: not all data is good data.
I expanded my dataset to 30 cybersecurity Q&A pairs, then deliberately broke 3 of them — one duplicate, one missing question, one missing answer — just to see if my cleanup step would actually catch them.
Building a dataset is like proofreading an essay before you submit it — not adding new ideas, just catching mistakes that would embarrass you later. Two checks catch the most common ones: any row missing a question or answer, and any exact duplicate.
missing = [i for i, item in enumerate(data) if not item["instruction"].strip() or not item["response"].strip()]seen = set()duplicates = []for i, item in enumerate(data): key = (item["instruction"].strip().lower(), item["response"].strip().lower()) if key in seen: duplicates.append(i) seen.add(key)
php
Total examples loaded: 33Missing instruction/response: 2 -> indices [31, 32]Duplicate: 1 -> indices [30]Dropped 3 bad examples -> 30 clean examples remain
All 3 planted problems got caught. A duplicate is like a question appearing twice on an exam — it doesn’t test anything new, it just inflates that topic’s importance. A blank field is worse: it’s an empty answer sheet graded as correct.
Once the 30 examples are clean, the rest is the same process already covered above — wrap each pair in the chat template, tokenize it, and apply loss masking so the model only trains on the assistant’s turn.
Running this across all 30 examples, I also checked how long each one turned out to be after tokenizing. This matters because every model has a maximum number of tokens it can process at once — feed it more, and the extra content just gets cut off, taking part of the answer with it.
Example 1: 62 tokensToken length distribution: min=49, max=74, avg=59
Here, “min,” “max,” and “avg” just mean: across all 30 examples, the shortest one came out to 49 tokens, the longest to 74, and the average length was 59. In the code, I set a cutoff of max_length=512 — meaning anything longer than 512 tokens would get cut off. Since even my longest example was only 74 tokens, everything comfortably fit under that cutoff, and nothing got truncated.
Last step: split the 30 clean examples 80/20–24 for training, 6 held back as a validation set.
shuffled = loaded.copy()random.shuffle(shuffled)split_idx = int(len(shuffled) * 0.8)train_data = shuffled[:split_idx]val_data = shuffled[split_idx:]
Shuffling first avoids accidentally grouping similar topics together before splitting. split_idx is just 80% of 30 — everything before it becomes training data, everything after becomes validation.
Why hold examples back? A model can ace training data just by memorizing it — like memorizing last year’s exam without understanding the subject. Validation data is unseen, so it reveals whether the model actually learned, or just memorized. A big gap between the two is called overfitting.
With just 6 validation examples, this is too small to trust — it’s here to show the workflow, not produce a real score.
By the end: 33 raw rows → 30 clean → tokenized and masked → split into 24 training / 6 validation examples. This is the exact shape a fine-tuning tool expects — the format we’ll actually train on next.
Fine-tuning is the general idea — taking a pretrained model and adjusting it further. Supervised fine-tuning (SFT) is one specific way to do that: using labeled examples where the correct answer is already known, like a student learning from worked examples with the solution shown. That’s exactly what the Day 3 dataset provides — instruction/response pairs with a known right answer — which is why SFT is the right fit here.
Supervised — every example already has the correct answer attached. Fine-tuning — instead of building a model from zero, we take an existing one (GPT-2) and push it toward a narrower skill.
The process is a loop: give the model an example, let it predict, compare that prediction against the real answer already sitting in the data, measure how far off it was, adjust the weights to close the gap. Repeat enough times and the gap shrinks.
There’s no separate answer key — the “correct answer” is just the next token already written in the text:
Instruction:What is a firewall?Response:A firewall is a network security device...
Take the point in that example right after “…is a”. The model doesn’t know what comes next — it just assigns a probability to every word it knows: maybe 20% to “network,” 15% to “computer,” and so on. But we already know the real answer, because it’s sitting right there in the training text: the actual next word is “network.” Loss is just a measure of how much confidence the model put on that one correct word — high confidence means low loss, low confidence means high loss. Gradient descent then adjusts the weights slightly so “network” scores a bit higher next time it sees this same context.
Load JSONL → Clean → Format as text → Tokenize + split → Load base model→ Baseline test → Train → Save → Reload → Compare
Each pair gets formatted into GPT-2’s plain-text pattern, since GPT-2 has no chat template:
def format_for_sft(example): return { "text": f"Instruction:\n{example['instruction']}\n\nResponse:\n{example['response']}" }
SFTTrainer tokenizes this internally and takes the cleaned dataset, already split 24 training / 6 validation:
trainer = SFTTrainer( model=model, args=training_args, train_dataset=train_dataset, eval_dataset=eval_dataset)
Prompt:
Instruction:What is a firewall?
Output:
Response:The firewall is the firewall connected to a computer. The firewall is a firewallthat's designed to prevent unauthorized access by the user of the computer...
Circular, no real definition — untouched GPT-2.
trainer.train()
logging_steps=5,eval_steps=10,
logging_steps=5 prints progress only every 5th step (5, 10, 15, ... 60) — 12 lines, not 60. eval_steps=10 runs validation even less often, every 10th step (10, 20, 30, 40, 50, 60), checking the model against the 6 held-out examples it never trains on.
Step 5: loss 3.547, accuracy 0.36Step 30: loss 1.839, accuracy 0.62Step 60: loss 1.838, accuracy 0.61
Training loss dropped from 3.55 to roughly 1.8. mean_token_accuracy — how often the model's top guess matched the real token — rose from 36% to over 60%. Validation loss, measured on the 6 examples never trained on, dropped more smoothly from 2.56 to 2.11, confirming this wasn't just memorization.
The two curves tell slightly different stories:
Every red dot is an average of the same 6 test questions — those never change. What changes is the model taking the test: it keeps training in between each check, so each time it’s a little more practiced than before. Same questions, better-trained model — that’s why the average score keeps dropping as training goes on. The blue line is different — it’s based on just one training example per step, so a hard one makes it spike and an easy one makes it dip, with nothing to smooth that out. That’s why the red line is the one to trust: it drops steadily, without the up-and-down the blue line shows.
A firewall is a network segmentation attack that attempts to intercept and recovercommunications between computers and networks...
The definition itself is off — calling a firewall an “attack” is backwards. But the phrasing is noticeably more security-flavored: network, intercept, malicious, vulnerabilities. The model picked up the vocabulary before the facts. With 30 examples and 60 steps, that’s about as far as it gets.
trainer.save_model(output_dir)tokenizer.save_pretrained(output_dir)
Once saved, it reloads for inference in seconds, not the ~70 minutes training took:
generator = pipeline( "text-generation", model="./gpt2-cybersecurity-sft", device="cpu")
This became a standalone script, test_model.py, for asking new questions without rerunning the whole pipeline — plus test_base_model.py, plain GPT-2 the same way for a direct before/after comparison.
Loss falling and accuracy rising, on both train and validation data, proves the mechanism works — the model measurably learned the instruction→response pattern on the cleaned dataset. Whether it learned anything true is a separate question, one only more data and more steps can answer.
Day 4 trained and saved a model. Day 5 tests it — the untouched base model and the fine-tuned one, same questions, same settings, and a real number instead of a guess.
Two models, loaded fresh
base_model = AutoModelForCausalLM.from_pretrained("gpt2")ft_model = AutoModelForCausalLM.from_pretrained("../day-04-supervised-fine-tuning/gpt2-cybersecurity-sft")
base_model is plain GPT-2, no training at all. ft_model is loaded from the folder Day 4 saved to — the version actually trained on the 30 cybersecurity examples.
Same prompt format for both
The fine-tuned model only ever practiced questions in one shape during Day 4 training:
Instruction:{question}Response:
A plain question with no labels wasn’t something it ever saw. This function wraps every question into that same shape before it goes to either model:
def build_prompt(question): return f"Instruction:\n{question}\n\nResponse:\n"
That one wrapped prompt then goes to both models — same input, no unfair advantage either way:
base_out = base_pipe(prompt, ...) # base_model answeringft_out = ft_pipe(prompt, ...) # ft_model answering
Left column is base_model's answers, right column is ft_model's. The difference shows up fast — asked to explain encryption, the base model's answer isn't really an answer at all, it drifts into what looks like a copied forum reply, even including someone's email address mid-sentence. The fine-tuned model doesn't do this anywhere — all five answers stay focused on the actual question asked.
What perplexity actually is
Imagine reading a sentence one word at a time, and after each word, pausing to guess what’s coming next.
Perplexity is that confusion, turned into a number. Roughly: at each point, how many different words did the model seriously think could come next?
Lower is better.
Where the real text actually gets shown to the model
Worth being clear here: base_model was never trained on anything — it's simply handed the validation text cold and scored on how well it predicts it. ft_model is the one that actually trained on this kind of data back in Day 4. Same test, same 6 examples, but one model is walking in blind and the other has studied.
The text is the same 6 validation examples from Day 3 — never used for training, formatted the same way:
return [f"Instruction:\n{ex['instruction']}\n\nResponse:\n{ex['response']}" for ex in val]
Each model reads these 6 examples, one at a time, and gets checked on how well it guessed each next word — same idea as Day 4, except nothing gets adjusted here. It’s just a test:
for text in texts: inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=256) outputs = model(**inputs, labels=inputs["input_ids"]) losses.append(outputs.loss.item())
Each example gets its own confusion score. After all 6, those scores get averaged, then turned into perplexity:
avg_loss = sum(losses) / len(losses)return math.exp(avg_loss), avg_loss
Runs twice — once for the base model, once for the fine-tuned one — same 6 examples both times:
base_ppl, base_loss = compute_perplexity(base_model, base_tokenizer, val_texts)ft_ppl, ft_loss = compute_perplexity(ft_model, ft_tokenizer, val_texts)
Base GPT-2 — perplexity: 38.42Fine-Tuned — perplexity: 5.33
Base GPT-2 was confused reading this text — like guessing among 38 possible next words each time. The fine-tuned model narrowed that down to about 5 — much more confident, since this is exactly what it trained on. Worth remembering though: this measures how naturally the wording flows, not whether it’s correct — the same fine-tuned model that scored 5.3 here is the one that confidently called a firewall an “attack” back in Day 4.
Before moving on, Week 2 gets one more piece: a small end-to-end project pulling every concept from these five days into a single pipeline — chat templates, loss masking, cleaning, SFT, evaluation — on a slightly bigger dataset than 30 examples.
Then, Week 3 — Efficient Fine-Tuning & Quantization
If you missed Week 1, it’s here:Understanding Large Language Models: From Neural Networks to Production Inference
No GPU required. Run everything on your laptop.
From Prompt Engineering to Fine-Tuning: Building Domain-Specific LLMs Step by Step was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.