{"slug": "unsloth-ollama-fine-tune-and-run-local-llms", "title": "Unsloth + Ollama: Fine-Tune and Run Local LLMs", "summary": "Unsloth and Ollama can be paired to fine-tune an 8B-parameter open-source LLM with QLoRA on a consumer GPU using under 8GB of VRAM and then serve it locally, according to a walkthrough published by Unsloth. The guide states Unsloth's hand-crafted OpenAI Triton kernels rewrite PyTorch attention layers and cross-entropy loss to deliver 2x faster training and 70% to 80% less VRAM, letting Llama 3 8B or Mistral 7B train in 6.6GB to 8GB of VRAM. The pipeline exports directly to quantized GGUF format, auto-generates an Ollama Modelfile, and is presented as completable in under 30 minutes.", "body_md": "# Unsloth + Ollama: Fine-Tune and Run Local LLMs\n\nUnsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can pe\n\nUnsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens. Fine-tuning an 8B parameter model no longer requires a cluster of enterprise A100 GPUs or endless cloud compute budgets. By pairing Unsloth with Ollama, you can perform QLoRA local fine tuning on a consumer GPU using under 8GB of VRAM, convert the model directly to GGUF format, and serve it on your local machine with a single CLI command. Here is how to set up the entire end-to-end pipeline in under 30 minutes. Standard Hugging Face pipelines often encounter CUDA Out-Of-Memory (OOM) crashes on 16GB or 24GB GPUs when training 8B parameter models. Unsloth rewrites PyTorch attention layers and cross-entropy loss functions into hand-crafted OpenAI Triton kernels. This optimization yields concrete advantages: 2x Faster Training: Bypasses PyTorch autograd overhead. 70% to 80% Less VRAM: Train Llama 3 8B or Mistral 7B in 6.6GB to 8GB VRAM with QLoRA. Zero Conversion Overhead: Exports directly to quantized GGUF format and auto-generates the Ollama Modelfile. Zero API Bills: Serve your proprietary tuned model locally on macOS, Linux, or Windows via Ollama. Before starting, bookmark and pull resources from the official repositories: Unsloth GitHub Repository: https://github.com/unslothai/unsloth — Core training framework and Triton kernels. Unsloth Official Documentation: https://unsloth.ai/docs — Setup guides, memory benchmarks, and architecture details. Unsloth Hugging Face Hub: https://huggingface.co/unsloth — Official 4-bit pre-quantized base models (Llama 3, Mistral, Qwen, Gemma). Ollama Download: https://ollama.com/download — Standalone installers for macOS, Linux, and Windows. Ollama GitHub Repository: https://github.com/ollama/ollama — Open-source local runtime repository. Set up Ollama and an isolated Python virtual environment on your Linux/WSL2 system with CUDA installed. curl -fsSL https://ollama.com/install.sh | sh Create a dedicated environment and install Unsloth alongside PyTorch dependencies: python3 -m venv finetune-env source finetune-env/bin/activate pip install --upgrade pip pip install 'unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git' pip install --no-deps trl peft accelerate bitsandbytes datasets In this walkthrough, we will fine-tune Llama 3 8B Instruct on a customer support dataset, export it to GGUF, and query it locally via Ollama. Create a training script named train.py. Load the pre-quantized 4-bit model directly from Hugging Face and inject LoRA parameters: import torch from unsloth import FastLanguageModel from datasets import load_dataset from trl import SFTTrainer from transformers import TrainingArguments max_seq_length = 2048 dtype = None # Auto-detect Float16 / Bfloat16 load_in_4bit = True # 1. Load 4-bit base model model, tokenizer = FastLanguageModel.from_pretrained( model_name = 'unsloth/llama-3-8b-Instruct-bnb-4bit', max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) # 2. Attach LoRA adapters model = FastLanguageModel.get_peft_model( model, r = 16, target_modules = ['q_proj', 'k_proj', 'v_proj', 'o_proj', 'gate_proj', 'up_proj', 'down_proj'], lora_alpha = 16, lora_dropout = 0, bias = 'none', use_gradient_checkpointing = 'unsloth', random_state = 3407, ) Train on instruction-response pairs using SFTTrainer for 60 steps: # 3. Format dataset using Llama 3 template prompt_template = '''<|start_header_id|>system<|end_header_id|> You are a specialized internal support bot.<|eot_id|><|start_header_id|>user<|end_header_id|> {}<|eot_id|><|start_header_id|>assistant<|end_header_id|> {}<|eot_id|>''' def formatting_prompts_func(examples): texts = [] for instruction, output in zip(examples['instruction'], examples['output']): text = prompt_template.format(instruction, output) texts.append(text) return {'text': texts} dataset = load_dataset('philschmid/dolly-15k-oai-style', split='train[:1000]') dataset = dataset.map(formatting_prompts_func, batched=True) trainer = SFTTrainer( model = model, tokenizer = tokenizer, train_dataset = dataset, dataset_text_field = 'text', max_seq_length = max_seq_length, dataset_num_proc = 2, packing = False, args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 60, learning_rate = 2e-4, fp16 = not torch.cuda.is_bf16_supported(), bf16 = torch.cuda.is_bf16_supported(), logging_steps = 10, optim = 'adamw_8bit', weight_decay = 0.01, lr_scheduler_type = 'linear', seed = 3407, output_dir = 'outputs', ), ) trainer.train() Unsloth eliminates manual llama.cpp compilation. Export directly to 4-bit Medium Quantization (q4_k_m) at the end of train.py: # 4. Export directly to GGUF model.save_pretrained_gguf('custom_model', tokenizer, quantization_method = 'q4_k_m') Run the training file: python train.py This writes the quantized GGUF weights and an automatically configured Modelfile into the ./custom_model directory. Point Ollama to the generated folder and launch the model: # 5. Build and serve inside Ollama ollama create my-assistant -f ./custom_model/Modelfile ollama run my-assistant Example Interaction: >>> How do I reset my company VPN token? To reset your company VPN token: 1. Open the Okta Verify dashboard on your mobile device. 2. Navigate to Settings > Enrolled Factors > VPN Gateway. 3. Click 'Regenerate Token' and confirm your biometric prompt. Your new session token will be active within 60 seconds. Combining Unsloth and Ollama creates a self-contained local fine-tuning workflow: train locally in 15–30 minutes, eliminate external API dependencies, and keep all proprietary data within your own infrastructure.\n\n## Key Takeaways\n\n- •Unsloth + Ollama: Fine-Tune Open-Source LLMs 2x Faster and Run Them Locally Stop burning thousands of dollars on cloud LLM API tokens\n- •This story was reported by **Dev.to** , covering developments in the**dev** space.\n- •AI advancements continue to reshape industries — read the full article on Dev.to for complete coverage.\n\n📖 Continue reading the full article:\n\n[Read Full Article on Dev.to →](https://dev.to/signalmatrixai/unsloth-ollama-fine-tune-and-run-local-llms-3p78)", "url": "https://wpnews.pro/news/unsloth-ollama-fine-tune-and-run-local-llms", "canonical_source": "https://ainexusdaily.vercel.app/article/2026-10-12-unsloth-ollama-fine-tune-and-run-local-llms", "published_at": "2026-10-12 14:36:13+00:00", "updated_at": "2026-10-12 14:46:53.164420+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["Unsloth", "Ollama", "Llama 3 8B Instruct", "Mistral 7B", "PyTorch", "OpenAI Triton", "Hugging Face", "QLoRA"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/unsloth-ollama-fine-tune-and-run-local-llms", "markdown": "https://wpnews.pro/news/unsloth-ollama-fine-tune-and-run-local-llms.md", "text": "https://wpnews.pro/news/unsloth-ollama-fine-tune-and-run-local-llms.txt", "jsonld": "https://wpnews.pro/news/unsloth-ollama-fine-tune-and-run-local-llms.jsonld"}}