{"slug": "how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on", "title": "How to Fine-Tune LFM2 Using QLoRA and DPO: A Complete Step-by-Step Coding Tutorial on Google Colab", "summary": "Liquid AI's LFM2 model can be fine-tuned using QLoRA and DPO in a complete open-source workflow on Google Colab. The process loads the base LFM2 checkpoint with 4-bit quantization, trains a lightweight LoRA adapter using TRL and PEFT, and extends the workflow with DPO to improve response preference using chosen and rejected answers. The resulting pipeline moves from a base LFM2 model to a supervised fine-tuned, preference-aligned checkpoint ready for testing or deployment.", "body_md": "In this tutorial, we fine-tune [ Liquid AI’s LFM2](https://github.com/Liquid4All/leap-finetune) model through a complete open-source workflow. We start by loading the base LFM2 checkpoint with QLoRA, preparing a chat-style supervised fine-tuning dataset, training a lightweight LoRA adapter using TRL and PEFT, and then merging the adapter back into the model. We also extend the workflow with DPO to show how we can improve response preference using chosen and rejected answers. At the end, we have a practical pipeline that moves from a base LFM2 model to an SFT-tuned, preference-aligned checkpoint, ready for further testing or deployment.\n\n```\n!pip install -q -U \"transformers>=4.55\" \"trl>=0.12\" \"peft>=0.13\" \"datasets>=2.20\" \"accelerate>=0.34\" bitsandbytes\n\nimport torch, gc\nfrom datasets import load_dataset, Dataset\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig\nfrom peft import LoraConfig, PeftModel, prepare_model_for_kbit_training\nfrom trl import SFTConfig, SFTTrainer, DPOConfig, DPOTrainer\n\nMODEL_ID    = \"LiquidAI/LFM2-1.2B\"\nUSE_4BIT    = True\nRUN_DPO     = True\nSFT_SAMPLES = 500\nSFT_STEPS   = 60\nDPO_STEPS   = 40\nMAX_LEN     = 1024\n\nBF16 = torch.cuda.is_available() and torch.cuda.is_bf16_supported()\nDTYPE = torch.bfloat16 if BF16 else torch.float16\nassert torch.cuda.is_available(), \"No GPU detected — set Runtime > Change runtime type > GPU\"\nprint(f\"GPU: {torch.cuda.get_device_name(0)} | dtype={DTYPE} | 4bit={USE_4BIT}\")\n```\n\nWe install all the required libraries for fine-tuning LFM2 inside Google Colab. We import the core tools from Transformers, TRL, PEFT, datasets, bitsandbytes, and PyTorch. We also define the main training settings, detect available GPUs, and select the appropriate precision for efficient training.\n\n``` python\ndef load_base(four_bit: bool):\n   quant_cfg = None\n   if four_bit:\n       quant_cfg = BitsAndBytesConfig(\n           load_in_4bit=True,\n           bnb_4bit_quant_type=\"nf4\",\n           bnb_4bit_use_double_quant=True,\n           bnb_4bit_compute_dtype=DTYPE,\n       )\n   model = AutoModelForCausalLM.from_pretrained(\n       MODEL_ID,\n       device_map=\"auto\",\n       dtype=DTYPE,\n       quantization_config=quant_cfg,\n   )\n   model.config.use_cache = False\n   return model\n\ntokenizer = AutoTokenizer.from_pretrained(MODEL_ID)\nif tokenizer.pad_token is None:\n   tokenizer.pad_token = tokenizer.eos_token\n\nmodel = load_base(USE_4BIT)\n\n@torch.no_grad()\ndef chat(m, user_msg, system=None, max_new_tokens=200):\n   msgs = ([{\"role\": \"system\", \"content\": system}] if system else []) + \\\n          [{\"role\": \"user\", \"content\": user_msg}]\n   inputs = tokenizer.apply_chat_template(\n       msgs,\n       add_generation_prompt=True,\n       return_tensors=\"pt\",\n       tokenize=True,\n       return_dict=True,\n   ).to(m.device)\n   m.config.use_cache = True\n   out = m.generate(\n       **inputs,\n       max_new_tokens=max_new_tokens, do_sample=True,\n       temperature=0.3, min_p=0.15, repetition_penalty=1.05,\n       pad_token_id=tokenizer.pad_token_id,\n   )\n   m.config.use_cache = False\n   prompt_len = inputs[\"input_ids\"].shape[-1]\n   return tokenizer.decode(out[0, prompt_len:], skip_special_tokens=True)\n\nPROBE = \"Explain what makes the LFM2 architecture good for on-device AI, in 2 sentences.\"\nprint(\"\\n=== BASELINE (before fine-tuning) ===\\n\", chat(model, PROBE))\n```\n\nWe load the LFM2 base model with optional 4-bit quantization to reduce GPU memory usage. We prepare the tokenizer, set the padding token, and define a chat function for testing model responses. We then run a baseline prompt to compare the model’s behavior before and after fine-tuning.\n\n```\nsft_ds = load_dataset(\"HuggingFaceTB/smoltalk\", \"all\", split=f\"train[:{SFT_SAMPLES}]\")\nsft_ds = sft_ds.select_columns([\"messages\"])\nprint(\"\\nSFT example messages:\", sft_ds[0][\"messages\"][:2])\n\nlora_sft = LoraConfig(\n   r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n   task_type=\"CAUSAL_LM\", target_modules=\"all-linear\",\n)\n\nsft_cfg = SFTConfig(\n   output_dir=\"outputs/sft/lfm2_demo\",\n   max_length=MAX_LEN,\n   per_device_train_batch_size=2,\n   gradient_accumulation_steps=4,\n   learning_rate=2e-5,\n   warmup_ratio=0.03,\n   lr_scheduler_type=\"cosine\",\n   max_steps=SFT_STEPS,\n   logging_steps=10,\n   save_strategy=\"no\",\n   gradient_checkpointing=True,\n   gradient_checkpointing_kwargs={\"use_reentrant\": False},\n   bf16=BF16, fp16=not BF16,\n   optim=\"paged_adamw_8bit\" if USE_4BIT else \"adamw_torch\",\n   packing=False,\n   report_to=\"none\",\n)\n\nsft_trainer = SFTTrainer(\n   model=model,\n   args=sft_cfg,\n   train_dataset=sft_ds,\n   peft_config=lora_sft,\n   processing_class=tokenizer,\n)\nsft_trainer.train()\nsft_trainer.save_model(\"outputs/sft/lfm2_adapter\")\nprint(\"\\n=== AFTER SFT ===\\n\", chat(sft_trainer.model, PROBE))\n```\n\nWe load a chat-formatted supervised fine-tuning dataset and keep only the messages column. We configure LoRA for lightweight adapter-based training and define the SFT training settings. We then train the model with SFT, save the LoRA adapter, and test the improved model response.\n\n```\ndel sft_trainer, model\ngc.collect(); torch.cuda.empty_cache()\n\nbase_fp16 = AutoModelForCausalLM.from_pretrained(MODEL_ID, device_map=\"auto\", dtype=DTYPE)\nsft_merged = PeftModel.from_pretrained(base_fp16, \"outputs/sft/lfm2_adapter\").merge_and_unload()\nsft_merged.save_pretrained(\"outputs/sft/lfm2_merged\")\ntokenizer.save_pretrained(\"outputs/sft/lfm2_merged\")\nprint(\"Merged SFT model saved -> outputs/sft/lfm2_merged\")\n```\n\nWe clear the earlier training objects from memory to free GPU resources. We reload the base LFM2 model in fp16 or bf16 and attach the trained SFT LoRA adapter. We then merge the adapter into the base model and save the merged SFT checkpoint for the next stage.\n\n```\nif RUN_DPO:\n   pref_rows = [\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Reply to a customer whose order is late.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"I'm sorry your order is delayed. I've checked your tracking and it will arrive within 2 days — here's a 10% credit for the inconvenience.\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"Orders are sometimes late. Please wait.\"}]},\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Summarize the benefit of edge AI in one line.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"Edge AI runs models locally, giving low latency, offline reliability, and stronger privacy.\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"Edge AI is AI on the edge of things and it is good.\"}]},\n       {\"prompt\":  [{\"role\": \"user\", \"content\": \"Decline a meeting politely.\"}],\n        \"chosen\":  [{\"role\": \"assistant\", \"content\": \"Thanks for the invite — I have a conflict then. Could we find another slot this week?\"}],\n        \"rejected\":[{\"role\": \"assistant\", \"content\": \"No.\"}]},\n   ] * 20\n   pref_ds = Dataset.from_list(pref_rows)\n\n   lora_dpo = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias=\"none\",\n                         task_type=\"CAUSAL_LM\", target_modules=\"all-linear\")\n   dpo_cfg = DPOConfig(\n       output_dir=\"outputs/dpo/lfm2_demo\",\n       per_device_train_batch_size=1,\n       gradient_accumulation_steps=4,\n       learning_rate=5e-6,\n       beta=0.1,\n       max_length=MAX_LEN,\n       max_prompt_length=512,\n       max_steps=DPO_STEPS,\n       logging_steps=10,\n       save_strategy=\"no\",\n       gradient_checkpointing=True,\n       gradient_checkpointing_kwargs={\"use_reentrant\": False},\n       bf16=BF16, fp16=not BF16,\n       report_to=\"none\",\n   )\n   dpo_trainer = DPOTrainer(\n       model=sft_merged,\n       ref_model=None,\n       args=dpo_cfg,\n       train_dataset=pref_ds,\n       processing_class=tokenizer,\n       peft_config=lora_dpo,\n   )\n   dpo_trainer.train()\n   final = dpo_trainer.model.merge_and_unload()\n   final.save_pretrained(\"outputs/final/lfm2_sft_dpo\")\n   tokenizer.save_pretrained(\"outputs/final/lfm2_sft_dpo\")\n   print(\"\\n=== AFTER SFT + DPO ===\\n\", chat(dpo_trainer.model, PROBE))\n   print(\"Final model saved -> outputs/final/lfm2_sft_dpo\")\n\nprint(\"\\nDone. Compare the BASELINE vs AFTER-SFT(+DPO) outputs above.\")\n```\n\nWe optionally run DPO using prompt-chosen-and-rejected response pairs. We configure another LoRA adapter for preference tuning and train the SFT-merged model with DPO. We finally merge the DPO adapter, save the final model checkpoint, and compare the result against earlier outputs.\n\nIn conclusion, we built a full fine-tuning pipeline for LFM2 using only open-source tools, including Transformers, TRL, PEFT, datasets, and bitsandbytes. We used QLoRA to make training efficient on Colab GPUs, applied supervised fine-tuning to chat-formatted data, merged the trained adapter into the base model, and optionally further improved the model through DPO. It gives us a clear view of how modern LLM fine-tuning works in practice, from loading the model to producing a final checkpoint that can be compared against the original baseline and prepared for deployment.\n\nCheck out the ** Codes with Notebook here. **Also, feel free to follow us on\n\n**and don’t forget to join our**[Twitter](https://x.com/intent/follow?screen_name=marktechpost)\n\n**and Subscribe to**\n\n[150k+ ML SubReddit](https://www.reddit.com/r/machinelearningnews/)**. Wait! are you on telegram?**\n\n[our Newsletter](https://www.aidevsignals.com/)\n\n[now you can join us on telegram as well.](https://t.me/machinelearningresearchnews)Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? [Connect with us](https://forms.gle/wbash1wF6efRj8G58)\n\nSana Hassan, a consulting intern at Marktechpost and dual-degree student at IIT Madras, is passionate about applying technology and AI to address real-world challenges. With a keen interest in solving practical problems, he brings a fresh perspective to the intersection of AI and real-life solutions.\n\n- Sana Hassan\n- Sana Hassan\n- Sana Hassan\n- Sana Hassan", "url": "https://wpnews.pro/news/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on", "canonical_source": "https://www.marktechpost.com/2026/06/02/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-tutorial-on-google-colab/", "published_at": "2026-06-03 00:51:10+00:00", "updated_at": "2026-06-03 00:58:55.760521+00:00", "lang": "en", "topics": ["large-language-models", "machine-learning", "artificial-intelligence", "generative-ai", "ai-tools"], "entities": ["Liquid AI", "LFM2", "QLoRA", "DPO", "TRL", "PEFT", "Google Colab", "Hugging Face"], "alternates": {"html": "https://wpnews.pro/news/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on", "markdown": "https://wpnews.pro/news/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on.md", "text": "https://wpnews.pro/news/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on.txt", "jsonld": "https://wpnews.pro/news/how-to-fine-tune-lfm2-using-qlora-and-dpo-a-complete-step-by-step-coding-on.jsonld"}}