# I Fine-Tuned an LLM. > Source: > Published: 2026-11-10 15:01:04+00:00 AI is all the hype now, whether you’re a software engineer using it for your projects, or a student trying to write your 2000-word essay; AI is all around us. A simple search on Google sends you back a summary of all the search results, helping you with the answer at a glance. Copilot rewrites all your mails on Outlook for you. If (and you must have) you have spent any time at all with LLMs now, you would notice how well they cope with human language. Most often, with simple queries and common subjects, LLMs answer very well. The LLMs we are communicating with and evaluating the performance of, as of 8th November, 2026, are very advanced, like Claude Opus 5, or GPT 6 Astra. These LLMs are incredibly heavy in size; for context, if you wanted to run these models on your own machine, you’d need multiple terabytes of VRAM on your GPU. You definitely can’t afford them! (I know I can’t). Well, what if I wanted to use AI for my use-case? I could buy credits for AI usage for my application, write a really good prompt, and use the same advanced model on my own machine at a fraction of the price. But in addition to the price would come latency issues. What if I wanted to skip this price and test the limits of an open-source model? Would it perform just as well as a frontier model? Probably not. Could I train it to do it? Not from scratch, but you might be able to guide it towards the right answer. While this post does not teach the golden standard of fine-tuning LLMs, I still hope you pick up a new thing or two, from one learner to another. My use-case is an end-to-end alignment pipeline running on a single free-tier Kaggle T4 GPU: taking a compact 1.5B base model through **Supervised Fine-Tuning (SFT)**, **Direct Preference Optimization (DPO)**, and a **chained SFT + DPO** architecture. Along the way, I hit almost every classical failure mode in modern fine-tuning: compute capability mismatches, vocabulary distribution collapse, and models generating endless lines of MathfMathfMathf and Chinese glyphs. Let’s dive into it. Fine-tuning is a technique that adapts a pre-trained model to a new task. In other words, the LLM has already been trained on a very large dataset on compatible GPU hardware, which constitutes the pre-training part. But fine-tuning involves teaching the model to adapt to a task-/domain-specific dataset, which further improves the performance of the base model for a specific task. Fine-tuning exists so that you don’t have to train models from scratch everytime you have a new use-case — you expose the model to a bunch of examples of what you want the LLM to learn, and train the model such that it generates the output you desire. For example, if you were to give a base model (open-source, e.g., Qwen/Qwen2.5–1.5B) the following your assignment that’s due tonight: ``` Write a summary of Hamlet. ``` It might respond with: ``` Write a summary of Macbeth.Write a summary of King Lear. ``` Because in the dataset used for pre-training (like the internet), test prompts frequently appear inside lists of assignment prompts. You might not get an A on your essay this time. “But what if I prompt engineer an open-source model to summarize the text I give it? Can’t I just do what I usually do with ChatGPT?” Yes, you can — provided you use the Instruct-tuned model, and not a raw base model. Open-source model families release two checkpoints: But open-source models are not big enough to accommodate heavy requests. Let’s demonstrate using a simple example. Suppose the task is to summarize the following text: “About a year and a half, maybe 2 years ago I started having back issues, mainly right between my shoulder blades. It basically feels like my spine is not aligned properly and I can’t help but fidget around. I stretch or bend and fidget in an attempt to re-align it, when I do this it cracks, allowing me a few seconds of relief followed by an uncontrollable urge to try and crack it again. It affects my sleeping, my gf is great about it, but it must be slightly annoying to have me fidgeting around all the time.” We can perform a simple experiment here where we prompt-engineer the open-source model by Qwen to summarize the text above. We give it two different prompts: one is simple and does not have very many constraints, while the second prompt introduces strict business rules. Notice how the model’s response degraded when the instruction started including constraints and strict business rules: this is **prompt fragility**, and small models with only 1.5B parameters such as the one we used suffer from it when generating output for heavier use cases. On top of prompt fragility, you face two production bottlenecks: This is why we fine-tune base models for dedicated tasks. Instead of negotiating with an instruct model through complex prompt engineering on every inference pass, fine-tuning hardcodes the exact input-output transformation you desire into the model’s weights. By training on thousands of clean examples where input text directly meets the desired output syntax: SUBREDDIT: … TITLE: … POST: … TL;DR: {summary} The token sequence \n\nTL;DR: **becomes the instruction itself**. There is zero prompt engineering involved, with a near-zero token overhead. Fine-tuning collapses the general probabilistic search space of an LLM into a dedicated, deterministic software function. Now that you are convinced on the why, I will briefly touch upon some methods involved in fine-tuning. Several sources state several more methods of fine-tuning, but I found the following few to be the most relevant & most commonly mentioned. In this process, every single parameter (w & b) of a pre-trained model is updated and retrained using a new dataset. In general fine-tuning methods, the first few layers of the model are frozen to retain most of the general knowledge the LLM might have (e.g., the first few layers could have knowledge on basic language grammar + what most words mean, and so freezing these first few layers helps us skip having to retrain the model on information it already knows). PEFT is a method used to combat the cons posed by full fine-tuning. PEFT focuses on freezing the original weights of the model so that they do not lead to catastrophic forgetting. Supervised fine-tuning involves using a labelled dataset to fine-tune the model. The dataset helps the model learn the desired syntax and structure for a given input. This method involves reinforcement-based learning to fine-tune the model. There are 2 main types of reinforcement learning involved in fine-tuning models: In my experiment, I settled on using SFT + DPO, two techniques commonly paired together for fine-tuning smaller open-sourced LLMs. Instead of treating them as abstract concepts, I wanted to set up a controlled experiment to see what each mechanism actually brings to the table. I decided to build a 4-way evaluation: To make this experiment actually happen within reasonable constraints, here are the building blocks I chose: For the data, I turned to the trl-lib/tldr-preference dataset on Hugging Face (derived from Reddit’s r/TLDR). It has around 117k rows of Reddit submissions where users posted a detailed story or dilemma, followed by multiple candidate TLDRs that other users rated as either “chosen” (good) or “rejected” (bad). This dataset is interesting because of the heavy variety in its samples. I also chose it because of its compatibility with the trl-lib library, which exposes DPOTrainer and SFTTrainer classes for fine-tuning models using these techniques! For fine-tuning, I picked the base model of Qwen 2.5. This model has about 1.5B parameters, which can smoothly run on the cloud GPU I used to fine-tune the model on Kaggle! I hope I’m not the only one who did not know about the free 30 hours of GPU / week that Kaggle offers to everyone, because that would be embarrassing. Regardless, I used this for fine-tuning my model with 16GB NVIDIA T4 GPUs. While Kaggle often provisions dual T4s (T4 x2), I decided to use only a single card. Running backpropagation on even a small model like Qwen/Qwen2.5-1B on a T4 GPU with 16GB VRAM would cause a “CUDA out of Memory” error. To understand why this happens, we need to do a bit of hardware math. Loading a 1.5-billion-parameter model in standard 16-bit precision (float16 or bfloat16) takes about 3 GB of VRAM (1.5 * 10⁹ * 2 bytes). That sounds small enough for a 16 GB T4, right? The trap is that **training is much more expensive than inference**. During full fine-tuning, the AdamW optimizer has to keep track of: That adds up to **16 to 18 bytes of VRAM per parameter**. Suddenly, our modest 1.5B model requires nearly 25 to 30 GB of GPU memory just to calculate a single gradient step, instantly killing our 16GB T4 session. To bypass this wall, I paired two techniques: **LoRA (Low-Rank Adaptation)** and **NF4 Quantization (QLoRA)**. In standard fine-tuning, every single weight matrix update follows this classic equation: Where: This update isn’t just calculated once; it is recalculated **every single training step**: Doing this across millions of numbers, step after step, causes the GPU memory to exhaust itself, causing the CUDA out of memory crash. LoRA solves this with a basic property of matrix multiplication: If you multiply a matrix of dimensions (d x r) by another matrix of dimensions (r x k), the resulting matrix has dimensions (d x k). So instead of computing the **ΔW** matrix directly, LoRA **freezes W₀ completely** and decomposes **ΔW** into two much smaller matrices, A & B: Where r is a tiny chosen number (the **rank**; in my notebook, I used r = 16): The difference is that the optimizer never touches **W₀**, nor does it store optimizer buffers for the full 16.7 million values. It calculates gradients, momentum, and variance **strictly for A and B** — a total of just **parameters (65536 * 2)**. That is a **99.2% reduction in trainable parameters** for that layer! Two key architectural design choices keep this stable: LoRA shrinks the optimizer states, but what about the base model itself? That is where **QLoRA** comes in. Using the bitsandbytes library, we compress the frozen base model from 16-bit floating points down to **4-bit NormalFloat (NF4)**. NF4 is 4-bit quantization tailored specifically for bell-curve weights. By loading the base weights in 4-bit (instead of 16-bit) and computing forward passes in 16-bit precision through our LoRA adapters. The 1.5B base model fits comfortably in under **3 GB of VRAM**. Training iterations complete in roughly **2 to 3 minutes** on a single T4 card without breaking a sweat. With our memory budget fully under control, we can finally look at how the data and training pipelines were wired together in code. The following code ran in a Kaggle notebook session with internet access enabled to pull weights from HF. Upon closer inspection, you will notice how some rows share the same prompt (like rows 0, 1, 2, 3). For simplicity, we will reduce the dataset so that there is only 1 row per prompt with 1 “chosen” value (of the highest order from the logic above) and 1 “rejected” value (of the lowest order from the logic above). This reduces the dataset to just about 14k rows now. While tokenization is an important step in fine-tuning LLMs, some libraries manage this automatically. The trainers we will be using in this notebook are SFTTrainer and DPOTrainer from trl-lib, and hence, tokenization is handled by the trainers themselves, eliminating the need to manually do this. We will still check for “sequence lengths” and ensure that neither the prompt, nor the chosen value for any row exceeds our model’s context limit or the GPU resource’s VRAM capacity. Since the stats above show that the max length of the prompts & chosen values are small enough, we don’t need to do any truncation. We finally arrive at our favourite step: training the model :) As mentioned above, We will evaluate all these 3 model’s outputs against the output from the baseline model (model with no fine-tuning) too. For test purposes, we will be using only 600 samples. For the SFT subset of the dataset, we rename the “summary” column to “completion”, which is more compatible with the SFTTrainer class and retain the 3 column names as is for DPOTrainer. Quantize the model into 4 bits using bitsandbytes in Python. Inject matrices A & B into attention layers via peft_config: Specifies the exact sub-layers where adapter matrices are attached: ["q_proj", "k_proj", "v_proj", "o_proj"]. **Why this combination?** Targeting q_proj and v_proj alone is minimal. Expanding to all four projections adapts the full attention block while leaving MLP feed-forward layers (gate_proj, up_proj, down_proj) frozen. The following helper function is our quality-control checkpoint. It enforces three simple rules during inference: I evaluated the performed of each model on some unseen examples. For prompt 1, following is the golden standard from the dataset: Every time i turn on my laptop, it shows the extra monitor instead of the monitor it was displaying before. Given below are the output of each model for the same prompt: For prompt 2, following is the golden standard from the dataset: Dropkick Murphys saved a guy’s life by having the crowd back up and letting the medics in to get him out. For prompt 3, following is the golden standard from the dataset: Government bureaucracy claims they sent me a letter, but I never got it. Why don’t some official places like that e-mail too? I would be lying if I said I got this code right on the first try. Before getting clean outputs, I hit nearly every classic pitfall. Here’s a brief summary of what issues I encountered, and how I fixed them: When setting up training arguments in Hugging Face, almost every modern guide tells you to use bf16=True. It prevents underflow, stabilizes gradients, and handles dynamic ranges much better than standard float16. So naturally, I turned it on. And the training immediately crashed. At one point early on, I prompted the model for a summary and received output that looked like this: *Baseline summary: MathfMathfMathf 很多时候 我们在处理...* A 1.5B model prompted with an English Reddit post suddenly collapsed into repeating obscure math tokens and spitting out Chinese characters! Even after the model learned how to write a good summary, it would occasionally add weird phrases at the end: *...A band saved a guy's life. EDIT: thanks for the gold kind stranger!* Fine-tuning small, open-source language models isn’t about competing with trillion-parameter frontier APIs on general knowledge or philosophy. It is about **domain specialization, resource ownership, and operational predictability**. Here is what this experiment validated: I hope this post inspired you to fine-tune your own LLM :) Feel free to correct me if I’m wrong about anything in this post! [I Fine-Tuned an LLM.](https://pub.towardsai.net/i-fine-tuned-an-llm-e1c9d79a7b90) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.