{"slug": "fine-tune-qwen2-5-7b-with-qlora-on-your-own-data", "title": "Fine-Tune Qwen2.5-7B with QLoRA on Your Own Data", "summary": "Mariana Souza published a practical guide for fine-tuning Qwen2.5-7B-Instruct using QLoRA on custom instruction datasets, including cost estimates and a loss-masking sanity check. The tutorial covers dataset preparation, 4-bit model loading, and merging for local or server deployment, with real-world compute costs under $10.", "body_md": "# Fine-Tune Qwen2.5-7B with QLoRA on Your Own Data\n\nA practical walkthrough for QLoRA fine-tuning Qwen2.5-7B-Instruct on a custom instruction dataset, with real cost numbers and a loss-masking sanity check most tutorials skip.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)\n\n## What you'll build\n\nTake a JSONL dataset of instruction/response pairs, fine-tune Qwen2.5-7B-Instruct with QLoRA (4-bit base, LoRA adapters), confirm the trainer masks loss on the right tokens, and end up with a merged model you can run locally or serve. Includes real numbers on time and cost, and a clear answer to \"should I even do this.\"\n\n## Prerequisites\n\n- Linux, NVIDIA GPU with 24GB+ VRAM (RTX 4090, L4, A10G, A100). QLoRA training at the batch size/sequence length used below runs roughly 14-18GB; leave headroom either way.\n- NVIDIA driver supporting CUDA 12.1+ (\n`nvidia-smi`\n\nshould show`CUDA Version: 12.1`\n\nor higher). - Python 3.10 or 3.11 in a fresh virtualenv.\n- A Hugging Face account (Qwen2.5-7B-Instruct is Apache-2.0, ungated, but you'll want an account for caching and rate limits).\n- 500-5,000 labeled examples in your domain. Fewer than a few hundred and LoRA usually underperforms good prompting; you're wasting a GPU.\n\nPin your stack exactly. TRL's `SFTConfig`\n\nhas moved fields around across releases, and version drift is the single biggest source of \"it silently trains wrong\" bug reports:\n\n```\npython -m venv .venv && source .venv/bin/activate\npip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu121\npip install transformers==4.46.3 trl==0.12.2 peft==0.13.2 \\\n bitsandbytes==0.44.1 accelerate==1.0.1 datasets==3.1.0\n```\n\n## Step 1: Decide if you actually need this\n\nFine-tuning fixes: consistent output format across thousands of calls, domain jargon the base model garbles, shorter prompts because instructions are baked into weights instead of repeated every call. It doesn't fix: knowledge that changes weekly (use RAG), reasoning the base model fundamentally lacks (more data won't add it), or a one-off task (few-shot prompting is cheaper and faster to iterate on).\n\nCost reality: QLoRA on a rented L4 or A10G runs roughly $0.50-1.20/hour on spot instances. A 2,000-example dataset for 3 epochs takes 1-3 hours depending on sequence length, so budget under $10 in compute. Data prep and eval cost more than the GPU time does. If you're reaching for full fine-tuning (all params, no LoRA) on anything above 7B, you're now talking multi-GPU and real money, don't default to it.\n\n## Step 2: Prepare the dataset\n\nUse the chat message format TRL expects. One JSON object per line:\n\n```\n{\"messages\": [{\"role\": \"system\", \"content\": \"You are a support agent for Acme Cloud.\"}, {\"role\": \"user\", \"content\": \"My deploy is stuck at pending.\"}, {\"role\": \"assistant\", \"content\": \"Check `kubectl describe pod` for the pending pod...\"}]}\n```\n\nSplit into `train.jsonl`\n\nand `val.jsonl`\n\n(90/10 is fine for a few thousand examples). Load and format:\n\n``` python\nfrom datasets import load_dataset\nfrom transformers import AutoTokenizer\n\nbase_model = \"Qwen/Qwen2.5-7B-Instruct\"\ntokenizer = AutoTokenizer.from_pretrained(base_model)\n\nraw = load_dataset(\"json\", data_files={\"train\": \"train.jsonl\", \"validation\": \"val.jsonl\"})\n\ndef format_example(example):\n return {\"text\": tokenizer.apply_chat_template(example[\"messages\"], tokenize=False)}\n\ndataset = raw.map(format_example, remove_columns=raw[\"train\"].column_names)\n```\n\n`apply_chat_template`\n\nwraps turns in Qwen's `<|im_start|>role\\n...<|im_end|>`\n\nformat, which you'll need for masking in Step 4.\n\n## Step 3: Load the base model in 4-bit\n\n``` python\nimport torch\nfrom transformers import AutoModelForCausalLM, BitsAndBytesConfig\nfrom peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model\n\nbnb_config = BitsAndBytesConfig(\n load_in_4bit=True,\n bnb_4bit_quant_type=\"nf4\",\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_use_double_quant=True,\n)\n\nmodel = AutoModelForCausalLM.from_pretrained(\n base_model,\n quantization_config=bnb_config,\n device_map=\"auto\",\n torch_dtype=torch.bfloat16,\n)\nmodel.config.use_cache = False # incompatible with gradient checkpointing; flip back for inference\n\nmodel = prepare_model_for_kbit_training(\n model,\n use_gradient_checkpointing=True,\n gradient_checkpointing_kwargs={\"use_reentrant\": False},\n)\n\n# Belt-and-suspenders: this is what lets gradients flow into the frozen,\n# quantized embeddings under checkpointing. PEFT wires it automatically,\n# but transformers/peft version drift has broken it silently before.\nmodel.enable_input_require_grads()\n\nlora_config = LoraConfig(\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias=\"none\",\n task_type=\"CAUSAL_LM\",\n target_modules=[\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\", \"gate_proj\", \"up_proj\", \"down_proj\"],\n)\nmodel = get_peft_model(model, lora_config)\n```\n\n`use_reentrant=False`\n\nis the non-reentrant checkpointing path; current PyTorch and Hugging Face guidance both recommend it over the reentrant default, which throws \"tensor does not require grad\" errors under PEFT more often. Since `prepare_model_for_kbit_training`\n\nalready turned checkpointing on, you'll set `gradient_checkpointing=False`\n\nin the trainer config below so it isn't fighting PEFT for the same mechanism.\n\n## Step 4: Configure the trainer\n\n**Version note**: `dataset_text_field`\n\n, `max_seq_length`\n\n, and `packing`\n\nare `SFTConfig`\n\nfields as of TRL 0.12.2. Later TRL releases reorganized this config (fields renamed or moved). If you didn't pin `trl==0.12.2`\n\n, these may error out or get silently ignored.\n\n``` python\nfrom trl import SFTConfig, SFTTrainer, DataCollatorForCompletionOnlyLM\n\nresponse_template = \"<|im_start|>assistant\\n\"\ncollator = DataCollatorForCompletionOnlyLM(response_template=response_template, tokenizer=tokenizer)\n\nsft_config = SFTConfig(\n output_dir=\"./qwen2.5-7b-support-lora\",\n dataset_text_field=\"text\", # TRL 0.12.x only\n max_seq_length=2048, # TRL 0.12.x only, reorganized in later versions\n packing=False,\n gradient_checkpointing=False, # already handled in Step 3\n per_device_train_batch_size=4,\n gradient_accumulation_steps=4,\n num_train_epochs=3,\n learning_rate=2e-4,\n lr_scheduler_type=\"cosine\",\n warmup_ratio=0.03,\n logging_steps=10,\n eval_strategy=\"steps\",\n eval_steps=50,\n save_strategy=\"steps\",\n save_steps=50,\n save_total_limit=2,\n bf16=True,\n report_to=\"none\",\n)\n\ntrainer = SFTTrainer(\n model=model,\n args=sft_config,\n train_dataset=dataset[\"train\"],\n eval_dataset=dataset[\"validation\"],\n data_collator=collator,\n tokenizer=tokenizer,\n)\n```\n\nThis setup masks correctly for single-turn examples, one assistant turn per example, like Step 2. For multi-turn data, also pass `instruction_template=\"<|im_start|>user\\n\"`\n\nso the collator re-masks each later user turn. Without it, only tokens before the *first* assistant turn get masked, everything after (including later user turns) leaks into the loss.\n\nOne more subtlety: `response_template`\n\nmatching works reliably here because `<|im_start|>`\n\nand `<|im_end|>`\n\nare real special tokens in Qwen's vocab, not regular BPE-merged text, so the template tokenizes identically whether it appears standalone or mid-sequence. That's not guaranteed for every chat format, which is exactly why Step 5 exists instead of trusting this by inspection.\n\nAlso: with transformers ≥4.46, passing `tokenizer=`\n\nto `SFTTrainer`\n\nthrows a deprecation warning in favor of `processing_class=`\n\n. It still works on these pinned versions, ignore it.\n\n## Step 5: Verify the loss mask before you burn GPU hours\n\nDon't trust masking by eyeballing the template string. Pull an actual batch through the trainer's own dataloader, since that's the exact path used during training:\n\n```\nbatch = next(iter(trainer.get_train_dataloader()))\nlabels = batch[\"labels\"][0]\ninput_ids = batch[\"input_ids\"][0]\n\nmasked = (labels == -100).sum().item()\ntotal = labels.numel()\nprint(f\"masked tokens: {masked}/{total}\")\n\nresponse_ids = input_ids[labels != -100]\nprint(tokenizer.decode(response_ids))\n```\n\nThe printed text should be *only* the assistant's response, no system prompt or user turn. If your data is multi-turn and you skipped the `instruction_template`\n\nfix, this is exactly where you'll catch it, a later user turn's text sitting inside the unmasked region.\n\nWhy this method and not a manual re-tokenize: hand-tokenizing a sample string with `tokenizer(text, add_special_tokens=False)`\n\nonly matches training exactly for tokenizers that add no BOS token, which Qwen's happens to be. For a tokenizer that does prepend BOS (Llama, Mistral), a hand-tokenized check would be off by one token from what the trainer actually sees. Going through `get_train_dataloader()`\n\nis ground truth regardless of tokenizer quirks.\n\n## Step 6: Train and evaluate\n\n```\ntrainer.train()\ntrainer.save_model(\"./qwen2.5-7b-support-lora/final\")\n```\n\nSaving explicitly to a `final`\n\ndirectory means you're not hunting through `checkpoint-50`\n\n, `checkpoint-100`\n\n, etc. afterward to figure out which one to merge.\n\nWatch train loss and eval loss together. Eval loss climbing while train loss keeps dropping past epoch 1-2 means overfitting, either cut epochs or add data. For a genuine read, don't trust loss alone: hold out 20-30 prompts, generate from both base and fine-tuned checkpoints, and read them side by side. Loss can look fine while the model still ignores your formatting requirements.\n\n## Step 7: Merge and use the model\n\nYou can't merge LoRA weights into a 4-bit base directly, you need the base in full precision:\n\n``` python\nfrom peft import PeftModel\n\nbase = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16, device_map=\"auto\")\nmerged = PeftModel.from_pretrained(base, \"./qwen2.5-7b-support-lora/final\").merge_and_unload()\nmerged.save_pretrained(\"./qwen2.5-7b-support-merged\")\ntokenizer.save_pretrained(\"./qwen2.5-7b-support-merged\")\n```\n\nIf you don't have headroom for the full-precision load plus merge on the same GPU, set `device_map=\"cpu\"`\n\ninstead, slower but no VRAM ceiling. Serve the result however you'd serve any local model: `vllm serve ./qwen2.5-7b-support-merged`\n\nor plain `transformers.generate`\n\n.\n\n## Verify it works\n\n`trainer.state.log_history`\n\nshould show eval loss trending down and stabilizing, not diverging.- The masking check in Step 5 should print clean assistant-only text with no system/user leakage.\n- Generate on 5-10 held-out prompts with the merged model and compare against base. You should see the fine-tuned model consistently matching your format/tone; if it looks identical to base, your learning rate or epoch count was too low, or your dataset is too small/noisy to move the model.\n\n## Troubleshooting\n\n**CUDA out of memory**: drop`per_device_train_batch_size`\n\nto 2, raise`gradient_accumulation_steps`\n\nto compensate, or reduce`max_seq_length`\n\n. QLoRA is memory-friendly, but 2048-token sequences at batch size 8 will still blow past 24GB.: gradient checkpointing wasn't wired to the input embeddings, or you're on the reentrant path. Confirm`element 0 of tensors does not require grad`\n\n`model.enable_input_require_grads()`\n\nran after model prep,`use_reentrant: False`\n\nis set, and`gradient_checkpointing=False`\n\nin`SFTConfig`\n\nso the trainer isn't fighting PEFT over the same mechanism.: you're on a newer TRL than 0.12.2. Either pin back to`SFTConfig`\n\nthrows`unexpected keyword argument 'max_seq_length'`\n\n`trl==0.12.2`\n\nor rewrite the config using that version's field names.**Loss is**: usually the learning rate is too high for LoRA (try`nan`\n\nafter a few steps`1e-4`\n\n), or you're on`fp16`\n\ninstead of`bf16`\n\non hardware that supports bf16, causing overflow.\n\n## Next steps\n\nOnce the adapter proves out, quantize the merged model (AWQ or GGUF) for cheaper serving, and build a real eval harness instead of eyeballing outputs, either `lm-evaluation-harness`\n\nfor standard benchmarks or a small rubric-based judge model for your specific task. If you need the model to prefer certain responses over others rather than just imitate a fixed target, look at DPO on top of this SFT checkpoint using the same TRL install.\n\n[Mariana Souza](https://sourcefeed.dev/u/mariana_souza)· Senior Editor\n\nMariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/fine-tune-qwen2-5-7b-with-qlora-on-your-own-data", "canonical_source": "https://sourcefeed.dev/a/fine-tune-qwen25-7b-with-qlora-on-your-own-data", "published_at": "2026-07-12 00:04:57+00:00", "updated_at": "2026-07-12 00:07:47.358486+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Mariana Souza", "Qwen2.5-7B-Instruct", "Hugging Face", "TRL", "PEFT", "bitsandbytes", "NVIDIA", "CUDA"], "alternates": {"html": "https://wpnews.pro/news/fine-tune-qwen2-5-7b-with-qlora-on-your-own-data", "markdown": "https://wpnews.pro/news/fine-tune-qwen2-5-7b-with-qlora-on-your-own-data.md", "text": "https://wpnews.pro/news/fine-tune-qwen2-5-7b-with-qlora-on-your-own-data.txt", "jsonld": "https://wpnews.pro/news/fine-tune-qwen2-5-7b-with-qlora-on-your-own-data.jsonld"}}