Fine-Tune Qwen2.5-7B with QLoRA on Your Own Data

Mariana Souza published a practical guide for fine-tuning Qwen2.5-7B-Instruct using QLoRA on custom instruction datasets, including cost estimates and a loss-masking sanity check. The tutorial covers dataset preparation, 4-bit model loading, and merging for local or server deployment, with real-world compute costs under $10.

Fine-Tune Qwen2.5-7B with QLoRA on Your Own Data A practical walkthrough for QLoRA fine-tuning Qwen2.5-7B-Instruct on a custom instruction dataset, with real cost numbers and a loss-masking sanity check most tutorials skip. Mariana Souza https://sourcefeed.dev/u/mariana souza What you'll build Take a JSONL dataset of instruction/response pairs, fine-tune Qwen2.5-7B-Instruct with QLoRA 4-bit base, LoRA adapters , confirm the trainer masks loss on the right tokens, and end up with a merged model you can run locally or serve. Includes real numbers on time and cost, and a clear answer to "should I even do this." Prerequisites - Linux, NVIDIA GPU with 24GB+ VRAM RTX 4090, L4, A10G, A100 . QLoRA training at the batch size/sequence length used below runs roughly 14-18GB; leave headroom either way. - NVIDIA driver supporting CUDA 12.1+ nvidia-smi should show CUDA Version: 12.1 or higher . - Python 3.10 or 3.11 in a fresh virtualenv. - A Hugging Face account Qwen2.5-7B-Instruct is Apache-2.0, ungated, but you'll want an account for caching and rate limits . - 500-5,000 labeled examples in your domain. Fewer than a few hundred and LoRA usually underperforms good prompting; you're wasting a GPU. Pin your stack exactly. TRL's SFTConfig has moved fields around across releases, and version drift is the single biggest source of "it silently trains wrong" bug reports: python -m venv .venv && source .venv/bin/activate pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu121 pip install transformers==4.46.3 trl==0.12.2 peft==0.13.2 \ bitsandbytes==0.44.1 accelerate==1.0.1 datasets==3.1.0 Step 1: Decide if you actually need this Fine-tuning fixes: consistent output format across thousands of calls, domain jargon the base model garbles, shorter prompts because instructions are baked into weights instead of repeated every call. It doesn't fix: knowledge that changes weekly use RAG , reasoning the base model fundamentally lacks more data won't add it , or a one-off task few-shot prompting is cheaper and faster to iterate on . Cost reality: QLoRA on a rented L4 or A10G runs roughly $0.50-1.20/hour on spot instances. A 2,000-example dataset for 3 epochs takes 1-3 hours depending on sequence length, so budget under $10 in compute. Data prep and eval cost more than the GPU time does. If you're reaching for full fine-tuning all params, no LoRA on anything above 7B, you're now talking multi-GPU and real money, don't default to it. Step 2: Prepare the dataset Use the chat message format TRL expects. One JSON object per line: {"messages": {"role": "system", "content": "You are a support agent for Acme Cloud."}, {"role": "user", "content": "My deploy is stuck at pending."}, {"role": "assistant", "content": "Check kubectl describe pod for the pending pod..."} } Split into train.jsonl and val.jsonl 90/10 is fine for a few thousand examples . Load and format: python from datasets import load dataset from transformers import AutoTokenizer base model = "Qwen/Qwen2.5-7B-Instruct" tokenizer = AutoTokenizer.from pretrained base model raw = load dataset "json", data files={"train": "train.jsonl", "validation": "val.jsonl"} def format example example : return {"text": tokenizer.apply chat template example "messages" , tokenize=False } dataset = raw.map format example, remove columns=raw "train" .column names apply chat template wraps turns in Qwen's <|im start| role\n...<|im end| format, which you'll need for masking in Step 4. Step 3: Load the base model in 4-bit python import torch from transformers import AutoModelForCausalLM, BitsAndBytesConfig from peft import prepare model for kbit training, LoraConfig, get peft model bnb config = BitsAndBytesConfig load in 4bit=True, bnb 4bit quant type="nf4", bnb 4bit compute dtype=torch.bfloat16, bnb 4bit use double quant=True, model = AutoModelForCausalLM.from pretrained base model, quantization config=bnb config, device map="auto", torch dtype=torch.bfloat16, model.config.use cache = False incompatible with gradient checkpointing; flip back for inference model = prepare model for kbit training model, use gradient checkpointing=True, gradient checkpointing kwargs={"use reentrant": False}, Belt-and-suspenders: this is what lets gradients flow into the frozen, quantized embeddings under checkpointing. PEFT wires it automatically, but transformers/peft version drift has broken it silently before. model.enable input require grads lora config = LoraConfig r=16, lora alpha=32, lora dropout=0.05, bias="none", task type="CAUSAL LM", target modules= "q proj", "k proj", "v proj", "o proj", "gate proj", "up proj", "down proj" , model = get peft model model, lora config use reentrant=False is the non-reentrant checkpointing path; current PyTorch and Hugging Face guidance both recommend it over the reentrant default, which throws "tensor does not require grad" errors under PEFT more often. Since prepare model for kbit training already turned checkpointing on, you'll set gradient checkpointing=False in the trainer config below so it isn't fighting PEFT for the same mechanism. Step 4: Configure the trainer Version note : dataset text field , max seq length , and packing are SFTConfig fields as of TRL 0.12.2. Later TRL releases reorganized this config fields renamed or moved . If you didn't pin trl==0.12.2 , these may error out or get silently ignored. python from trl import SFTConfig, SFTTrainer, DataCollatorForCompletionOnlyLM response template = "<|im start| assistant\n" collator = DataCollatorForCompletionOnlyLM response template=response template, tokenizer=tokenizer sft config = SFTConfig output dir="./qwen2.5-7b-support-lora", dataset text field="text", TRL 0.12.x only max seq length=2048, TRL 0.12.x only, reorganized in later versions packing=False, gradient checkpointing=False, already handled in Step 3 per device train batch size=4, gradient accumulation steps=4, num train epochs=3, learning rate=2e-4, lr scheduler type="cosine", warmup ratio=0.03, logging steps=10, eval strategy="steps", eval steps=50, save strategy="steps", save steps=50, save total limit=2, bf16=True, report to="none", trainer = SFTTrainer model=model, args=sft config, train dataset=dataset "train" , eval dataset=dataset "validation" , data collator=collator, tokenizer=tokenizer, This setup masks correctly for single-turn examples, one assistant turn per example, like Step 2. For multi-turn data, also pass instruction template="<|im start| user\n" so the collator re-masks each later user turn. Without it, only tokens before the first assistant turn get masked, everything after including later user turns leaks into the loss. One more subtlety: response template matching works reliably here because <|im start| and <|im end| are real special tokens in Qwen's vocab, not regular BPE-merged text, so the template tokenizes identically whether it appears standalone or mid-sequence. That's not guaranteed for every chat format, which is exactly why Step 5 exists instead of trusting this by inspection. Also: with transformers ≥4.46, passing tokenizer= to SFTTrainer throws a deprecation warning in favor of processing class= . It still works on these pinned versions, ignore it. Step 5: Verify the loss mask before you burn GPU hours Don't trust masking by eyeballing the template string. Pull an actual batch through the trainer's own dataloader, since that's the exact path used during training: batch = next iter trainer.get train dataloader labels = batch "labels" 0 input ids = batch "input ids" 0 masked = labels == -100 .sum .item total = labels.numel print f"masked tokens: {masked}/{total}" response ids = input ids labels = -100 print tokenizer.decode response ids The printed text should be only the assistant's response, no system prompt or user turn. If your data is multi-turn and you skipped the instruction template fix, this is exactly where you'll catch it, a later user turn's text sitting inside the unmasked region. Why this method and not a manual re-tokenize: hand-tokenizing a sample string with tokenizer text, add special tokens=False only matches training exactly for tokenizers that add no BOS token, which Qwen's happens to be. For a tokenizer that does prepend BOS Llama, Mistral , a hand-tokenized check would be off by one token from what the trainer actually sees. Going through get train dataloader is ground truth regardless of tokenizer quirks. Step 6: Train and evaluate trainer.train trainer.save model "./qwen2.5-7b-support-lora/final" Saving explicitly to a final directory means you're not hunting through checkpoint-50 , checkpoint-100 , etc. afterward to figure out which one to merge. Watch train loss and eval loss together. Eval loss climbing while train loss keeps dropping past epoch 1-2 means overfitting, either cut epochs or add data. For a genuine read, don't trust loss alone: hold out 20-30 prompts, generate from both base and fine-tuned checkpoints, and read them side by side. Loss can look fine while the model still ignores your formatting requirements. Step 7: Merge and use the model You can't merge LoRA weights into a 4-bit base directly, you need the base in full precision: python from peft import PeftModel base = AutoModelForCausalLM.from pretrained base model, torch dtype=torch.bfloat16, device map="auto" merged = PeftModel.from pretrained base, "./qwen2.5-7b-support-lora/final" .merge and unload merged.save pretrained "./qwen2.5-7b-support-merged" tokenizer.save pretrained "./qwen2.5-7b-support-merged" If you don't have headroom for the full-precision load plus merge on the same GPU, set device map="cpu" instead, slower but no VRAM ceiling. Serve the result however you'd serve any local model: vllm serve ./qwen2.5-7b-support-merged or plain transformers.generate . Verify it works trainer.state.log history should show eval loss trending down and stabilizing, not diverging.- The masking check in Step 5 should print clean assistant-only text with no system/user leakage. - Generate on 5-10 held-out prompts with the merged model and compare against base. You should see the fine-tuned model consistently matching your format/tone; if it looks identical to base, your learning rate or epoch count was too low, or your dataset is too small/noisy to move the model. Troubleshooting CUDA out of memory : drop per device train batch size to 2, raise gradient accumulation steps to compensate, or reduce max seq length . QLoRA is memory-friendly, but 2048-token sequences at batch size 8 will still blow past 24GB.: gradient checkpointing wasn't wired to the input embeddings, or you're on the reentrant path. Confirm element 0 of tensors does not require grad model.enable input require grads ran after model prep, use reentrant: False is set, and gradient checkpointing=False in SFTConfig so the trainer isn't fighting PEFT over the same mechanism.: you're on a newer TRL than 0.12.2. Either pin back to SFTConfig throws unexpected keyword argument 'max seq length' trl==0.12.2 or rewrite the config using that version's field names. Loss is : usually the learning rate is too high for LoRA try nan after a few steps 1e-4 , or you're on fp16 instead of bf16 on hardware that supports bf16, causing overflow. Next steps Once the adapter proves out, quantize the merged model AWQ or GGUF for cheaper serving, and build a real eval harness instead of eyeballing outputs, either lm-evaluation-harness for standard benchmarks or a small rubric-based judge model for your specific task. If you need the model to prefer certain responses over others rather than just imitate a fixed target, look at DPO on top of this SFT checkpoint using the same TRL install. Mariana Souza https://sourcefeed.dev/u/mariana souza · Senior Editor Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon. Discussion 0 No comments yet Be the first to weigh in.