Unsloth + Ollama: Fine-Tune and Run Local LLMs

Unsloth and Ollama can be paired to fine-tune an 8B-parameter open-source LLM with QLoRA on a consumer GPU using under 8GB of VRAM and then serve it locally, according to a walkthrough published by Unsloth. The guide states Unsloth's hand-crafted OpenAI Triton kernels rewrite PyTorch attention layers and cross-entropy loss to deliver 2x faster training and 70% to 80% less VRAM, letting Llama 3 8B or Mistral 7B train in 6.6GB to 8GB of VRAM. The pipeline exports directly to quantized GGUF format, auto-generates an Ollama Modelfile, and is presented as completable in under 30 minutes.

Unsloth + Ollama: Fine-Tune and Run Local LLMs Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can pe Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can perform QLoRA local fine tuning on a consumer GPU using under 8GB of VRAM, convert the model directly to GGUF format, and serve it on your local machine with a single CLI command. Here is how to set up the entire end-to-end pipeline in under 30 minutes. Standard Hugging Face pipelines often encounter CUDA Out-Of-Memory OOM crashes on 16GB or 24GB GPUs when training 8B parameter models. Unsloth rewrites PyTorch attention layers and cross-entropy loss functions into hand-crafted OpenAI Triton kernels. This optimization yields concrete advantages: 2x Faster Training: Bypasses PyTorch autograd overhead. 70% to 80% Less VRAM: Train Llama 3 8B or Mistral 7B in 6.6GB to 8GB VRAM with QLoRA. Zero Conversion Overhead: Exports directly to quantized GGUF format and auto-generates the Ollama Modelfile. Zero API Bills: Serve your proprietary tuned model locally on macOS, Linux, or Windows via Ollama. Before starting, bookmark and pull resources from the official repositories: Unsloth GitHub Repository: https://github.com/unslothai/unsloth — Core training framework and Triton kernels. Unsloth Official Documentation: https://unsloth.ai/docs — Setup guides, memory benchmarks, and architecture details. Unsloth Hugging Face Hub: https://huggingface.co/unsloth — Official 4-bit pre-quantized base models Llama 3, Mistral, Qwen, Gemma . Ollama Download: https://ollama.com/download — Standalone installers for macOS, Linux, and Windows. Ollama GitHub Repository: https://github.com/ollama/ollama — Open-source local runtime repository. Set up Ollama and an isolated Python virtual environment on your Linux/WSL2 system with CUDA installed. curl -fsSL https://ollama.com/install.sh | sh Create a dedicated environment and install Unsloth alongside PyTorch dependencies: python3 -m venv finetune-env source finetune-env/bin/activate pip install --upgrade pip pip install 'unsloth colab-new @ git+https://github.com/unslothai/unsloth.git' pip install --no-deps trl peft accelerate bitsandbytes datasets In this walkthrough, we will fine-tune Llama 3 8B Instruct on a customer support dataset, export it to GGUF, and query it locally via Ollama. Create a training script named train.py. Load the pre-quantized 4-bit model directly from Hugging Face and inject LoRA parameters: import torch from unsloth import FastLanguageModel from datasets import load dataset from trl import SFTTrainer from transformers import TrainingArguments max seq length = 2048 dtype = None Auto-detect Float16 / Bfloat16 load in 4bit = True 1. Load 4-bit base model model, tokenizer = FastLanguageModel.from pretrained model name = 'unsloth/llama-3-8b-Instruct-bnb-4bit', max seq length = max seq length, dtype = dtype, load in 4bit = load in 4bit, 2. Attach LoRA adapters model = FastLanguageModel.get peft model model, r = 16, target modules = 'q proj', 'k proj', 'v proj', 'o proj', 'gate proj', 'up proj', 'down proj' , lora alpha = 16, lora dropout = 0, bias = 'none', use gradient checkpointing = 'unsloth', random state = 3407, Train on instruction-response pairs using SFTTrainer for 60 steps: 3. Format dataset using Llama 3 template prompt template = '''<|start header id| system<|end header id| You are a specialized internal support bot.<|eot id| <|start header id| user<|end header id| {}<|eot id| <|start header id| assistant<|end header id| {}<|eot id| ''' def formatting prompts func examples : texts = for instruction, output in zip examples 'instruction' , examples 'output' : text = prompt template.format instruction, output texts.append text return {'text': texts} dataset = load dataset 'philschmid/dolly-15k-oai-style', split='train :1000 ' dataset = dataset.map formatting prompts func, batched=True trainer = SFTTrainer model = model, tokenizer = tokenizer, train dataset = dataset, dataset text field = 'text', max seq length = max seq length, dataset num proc = 2, packing = False, args = TrainingArguments per device train batch size = 2, gradient accumulation steps = 4, warmup steps = 5, max steps = 60, learning rate = 2e-4, fp16 = not torch.cuda.is bf16 supported , bf16 = torch.cuda.is bf16 supported , logging steps = 10, optim = 'adamw 8bit', weight decay = 0.01, lr scheduler type = 'linear', seed = 3407, output dir = 'outputs', , trainer.train Unsloth eliminates manual llama.cpp compilation. Export directly to 4-bit Medium Quantization q4 k m at the end of train.py: 4. Export directly to GGUF model.save pretrained gguf 'custom model', tokenizer, quantization method = 'q4 k m' Run the training file: python train.py This writes the quantized GGUF weights and an automatically configured Modelfile into the ./custom model directory. Point Ollama to the generated folder and launch the model: 5. Build and serve inside Ollama ollama create my-assistant -f ./custom model/Modelfile ollama run my-assistant Example Interaction: How do I reset my company VPN token? To reset your company VPN token: 1. Open the Okta Verify dashboard on your mobile device. 2. Navigate to Settings Enrolled Factors VPN Gateway. 3. Click 'Regenerate Token' and confirm your biometric prompt. Your new session token will be active within 60 seconds. Combining Unsloth and Ollama creates a self-contained local fine-tuning workflow: train locally in 15–30 minutes, eliminate external API dependencies, and keep all proprietary data within your own infrastructure. Key Takeaways - •Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens - •This story was reported by Dev.to , covering developments in the dev space. - •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage. 📖 Continue reading the full article: Read Full Article on Dev.to → https://dev.to/signalmatrixai/unsloth-ollama-fine-tune-and-run-local-llms-3p78