A practical walkthrough for QLoRA fine-tuning Qwen2.5-7B-Instruct on a custom instruction dataset, with real cost numbers and a loss-masking sanity check most tutorials skip.
What you'll build #
Take a JSONL dataset of instruction/response pairs, fine-tune Qwen2.5-7B-Instruct with QLoRA (4-bit base, LoRA adapters), confirm the trainer masks loss on the right tokens, and end up with a merged model you can run locally or serve. Includes real numbers on time and cost, and a clear answer to "should I even do this."
Prerequisites #
- Linux, NVIDIA GPU with 24GB+ VRAM (RTX 4090, L4, A10G, A100). QLoRA training at the batch size/sequence length used below runs roughly 14-18GB; leave headroom either way.
- NVIDIA driver supporting CUDA 12.1+ (
nvidia-smi
should showCUDA Version: 12.1
or higher). - Python 3.10 or 3.11 in a fresh virtualenv.
- A Hugging Face account (Qwen2.5-7B-Instruct is Apache-2.0, ungated, but you'll want an account for caching and rate limits).
- 500-5,000 labeled examples in your domain. Fewer than a few hundred and LoRA usually underperforms good prompting; you're wasting a GPU.
Pin your stack exactly. TRL's SFTConfig
has moved fields around across releases, and version drift is the single biggest source of "it silently trains wrong" bug reports:
python -m venv .venv && source .venv/bin/activate
pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.46.3 trl==0.12.2 peft==0.13.2 \
bitsandbytes==0.44.1 accelerate==1.0.1 datasets==3.1.0
Step 1: Decide if you actually need this #
Fine-tuning fixes: consistent output format across thousands of calls, domain jargon the base model garbles, shorter prompts because instructions are baked into weights instead of repeated every call. It doesn't fix: knowledge that changes weekly (use RAG), reasoning the base model fundamentally lacks (more data won't add it), or a one-off task (few-shot prompting is cheaper and faster to iterate on).
Cost reality: QLoRA on a rented L4 or A10G runs roughly $0.50-1.20/hour on spot instances. A 2,000-example dataset for 3 epochs takes 1-3 hours depending on sequence length, so budget under $10 in compute. Data prep and eval cost more than the GPU time does. If you're reaching for full fine-tuning (all params, no LoRA) on anything above 7B, you're now talking multi-GPU and real money, don't default to it.
Step 2: Prepare the dataset #
Use the chat message format TRL expects. One JSON object per line:
{"messages": [{"role": "system", "content": "You are a support agent for Acme Cloud."}, {"role": "user", "content": "My deploy is stuck at pending."}, {"role": "assistant", "content": "Check `kubectl describe pod` for the pending pod..."}]}
Split into train.jsonl
and val.jsonl
(90/10 is fine for a few thousand examples). Load and format:
from datasets import load_dataset
from transformers import AutoTokenizer
base_model = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model)
raw = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})
def format_example(example):
return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}
dataset = raw.map(format_example, remove_columns=raw["train"].column_names)
apply_chat_template
wraps turns in Qwen's <|im_start|>role\n...<|im_end|>
format, which you'll need for masking in Step 4.
Step 3: Load the base model in 4-bit #
import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
model = AutoModelForCausalLM.from_pretrained(
base_model,
quantization_config=bnb_config,
device_map="auto",
torch_dtype=torch.bfloat16,
)
model.config.use_cache = False # incompatible with gradient checkpointing; flip back for inference
model = prepare_model_for_kbit_training(
model,
use_gradient_checkpointing=True,
gradient_checkpointing_kwargs={"use_reentrant": False},
)
model.enable_input_require_grads()
lora_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora_config)
use_reentrant=False
is the non-reentrant checkpointing path; current PyTorch and Hugging Face guidance both recommend it over the reentrant default, which throws "tensor does not require grad" errors under PEFT more often. Since prepare_model_for_kbit_training
already turned checkpointing on, you'll set gradient_checkpointing=False
in the trainer config below so it isn't fighting PEFT for the same mechanism.
Step 4: Configure the trainer #
Version note: dataset_text_field
, max_seq_length
, and packing
are SFTConfig
fields as of TRL 0.12.2. Later TRL releases reorganized this config (fields renamed or moved). If you didn't pin trl==0.12.2
, these may error out or get silently ignored.
from trl import SFTConfig, SFTTrainer, DataCollatorForCompletionOnlyLM
response_template = "<|im_start|>assistant\n"
collator = DataCollatorForCompletionOnlyLM(response_template=response_template, tokenizer=tokenizer)
sft_config = SFTConfig(
output_dir="./qwen2.5-7b-support-lora",
dataset_text_field="text", # TRL 0.12.x only
max_seq_length=2048, # TRL 0.12.x only, reorganized in later versions
packing=False,
gradient_checkpointing=False, # already handled in Step 3
per_device_train_batch_size=4,
gradient_accumulation_steps=4,
num_train_epochs=3,
learning_rate=2e-4,
lr_scheduler_type="cosine",
warmup_ratio=0.03,
logging_steps=10,
eval_strategy="steps",
eval_steps=50,
save_strategy="steps",
save_steps=50,
save_total_limit=2,
bf16=True,
report_to="none",
)
trainer = SFTTrainer(
model=model,
args=sft_config,
train_dataset=dataset["train"],
eval_dataset=dataset["validation"],
data_collator=collator,
tokenizer=tokenizer,
)
This setup masks correctly for single-turn examples, one assistant turn per example, like Step 2. For multi-turn data, also pass instruction_template="<|im_start|>user\n"
so the collator re-masks each later user turn. Without it, only tokens before the first assistant turn get masked, everything after (including later user turns) leaks into the loss.
One more subtlety: response_template
matching works reliably here because <|im_start|>
and <|im_end|>
are real special tokens in Qwen's vocab, not regular BPE-merged text, so the template tokenizes identically whether it appears standalone or mid-sequence. That's not guaranteed for every chat format, which is exactly why Step 5 exists instead of trusting this by inspection.
Also: with transformers ≥4.46, passing tokenizer=
to SFTTrainer
throws a deprecation warning in favor of processing_class=
. It still works on these pinned versions, ignore it.
Step 5: Verify the loss mask before you burn GPU hours #
Don't trust masking by eyeballing the template string. Pull an actual batch through the trainer's own data, since that's the exact path used during training:
batch = next(iter(trainer.get_train_data()))
labels = batch["labels"][0]
input_ids = batch["input_ids"][0]
masked = (labels == -100).sum().item()
total = labels.numel()
print(f"masked tokens: {masked}/{total}")
response_ids = input_ids[labels != -100]
print(tokenizer.decode(response_ids))
The printed text should be only the assistant's response, no system prompt or user turn. If your data is multi-turn and you skipped the instruction_template
fix, this is exactly where you'll catch it, a later user turn's text sitting inside the unmasked region.
Why this method and not a manual re-tokenize: hand-tokenizing a sample string with tokenizer(text, add_special_tokens=False)
only matches training exactly for tokenizers that add no BOS token, which Qwen's happens to be. For a tokenizer that does prepend BOS (Llama, Mistral), a hand-tokenized check would be off by one token from what the trainer actually sees. Going through get_train_data()
is ground truth regardless of tokenizer quirks.
Step 6: Train and evaluate #
trainer.train()
trainer.save_model("./qwen2.5-7b-support-lora/final")
Saving explicitly to a final
directory means you're not hunting through checkpoint-50
, checkpoint-100
, etc. afterward to figure out which one to merge.
Watch train loss and eval loss together. Eval loss climbing while train loss keeps dropping past epoch 1-2 means overfitting, either cut epochs or add data. For a genuine read, don't trust loss alone: hold out 20-30 prompts, generate from both base and fine-tuned checkpoints, and read them side by side. Loss can look fine while the model still ignores your formatting requirements.
Step 7: Merge and use the model #
You can't merge LoRA weights into a 4-bit base directly, you need the base in full precision:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16, device_map="auto")
merged = PeftModel.from_pretrained(base, "./qwen2.5-7b-support-lora/final").merge_and_unload()
merged.save_pretrained("./qwen2.5-7b-support-merged")
tokenizer.save_pretrained("./qwen2.5-7b-support-merged")
If you don't have headroom for the full-precision load plus merge on the same GPU, set device_map="cpu"
instead, slower but no VRAM ceiling. Serve the result however you'd serve any local model: vllm serve ./qwen2.5-7b-support-merged
or plain transformers.generate
.
Verify it works #
trainer.state.log_history
should show eval loss trending down and stabilizing, not diverging.- The masking check in Step 5 should print clean assistant-only text with no system/user leakage.
- Generate on 5-10 held-out prompts with the merged model and compare against base. You should see the fine-tuned model consistently matching your format/tone; if it looks identical to base, your learning rate or epoch count was too low, or your dataset is too small/noisy to move the model.
Troubleshooting #
CUDA out of memory: dropper_device_train_batch_size
to 2, raisegradient_accumulation_steps
to compensate, or reducemax_seq_length
. QLoRA is memory-friendly, but 2048-token sequences at batch size 8 will still blow past 24GB.: gradient checkpointing wasn't wired to the input embeddings, or you're on the reentrant path. Confirmelement 0 of tensors does not require grad
model.enable_input_require_grads()
ran after model prep,use_reentrant: False
is set, andgradient_checkpointing=False
inSFTConfig
so the trainer isn't fighting PEFT over the same mechanism.: you're on a newer TRL than 0.12.2. Either pin back toSFTConfig
throwsunexpected keyword argument 'max_seq_length'
trl==0.12.2
or rewrite the config using that version's field names.Loss is: usually the learning rate is too high for LoRA (trynan
after a few steps1e-4
), or you're onfp16
instead ofbf16
on hardware that supports bf16, causing overflow.
Next steps #
Once the adapter proves out, quantize the merged model (AWQ or GGUF) for cheaper serving, and build a real eval harness instead of eyeballing outputs, either lm-evaluation-harness
for standard benchmarks or a small rubric-based judge model for your specific task. If you need the model to prefer certain responses over others rather than just imitate a fixed target, look at DPO on top of this SFT checkpoint using the same TRL install.
Mariana Souza· Senior Editor
Mariana covers the fast-moving world of machine learning and generative AI, with a particular focus on how these technologies are reshaping development workflows. When she isn't stress-testing the latest foundation models, she's usually at a local hackathon.
Discussion 0 #
No comments yet
Be the first to weigh in.