Beyondollama pull
— download any model from Hugging Face, quantize it, customize it, and import it into Ollama.
What's GGUF? #
GGUF (GPT-Generated Unified Format) is the standard file format for running LLMs locally. Think of it as the .mp3
of AI models:
Compressed— 70-85% smaller than the original float16 weights - Fast— optimized for CPU and GPU inference - Portable— one file contains the entire model - Metadata-rich— includes tokenizer, chat template, and model config
Every ollama pull
downloads a GGUF file under the hood. But the real power move is down GGUF files directly from Hugging Face and importing them yourself.
Quantization Analogy (Steal This)
Quantization is like
JPEG compression for AI models. A RAW photo is 50MB. A JPEG of the same photo is 5MB — 90% smaller, but it still looks 95% as good. That's what Q4_K_M quantization does to a model: 70% smaller, 96% of the intelligence.
Step 1: Finding the Right GGUF File #
The Golden Rule
Always look for Q4_K_M — it's the sweet spot of size vs quality for almost every model.
Where to Find GGUFs
| Source | URL | Best For |
|---|---|---|
| Official provider | ||
huggingface.co/Qwen etc. |
||
| Trustworthy, but often only Q8/Q6 | ||
| Unsloth | ||
huggingface.co/unsloth |
||
| Best selection of quants (Q2-Q8) | ||
| Bartowski | ||
huggingface.co/bartowski |
||
| Massive library, every quantization | ||
| MaziyarPanahi | ||
huggingface.co/MaziyarPanahi |
||
| Merged models, niche architectures |
The GGUF Filename Decoder
Qwen2.5-14B-Q4_K_M.gguf
├── Model name ├── Size └── Quantization
| Quant Code | Compression | Quality | Use Case |
|---|---|---|---|
| Q8_0 | 50% | 99% | When you have VRAM to spare |
| Q6_K | 60% | 98% | High-quality, reasonable size |
| Q4_K_M | |||
| 70% | |||
| 96% | |||
| 🟢 Sweet spot — use this | |||
| Q3_K_M | 78% | 92% | When VRAM is tight |
| Q2_K | 85% | 85% | Emergency only — quality noticeably drops |
| IQ4_XS | 72% | 95% | Experimental import format |
Step 2: Download & Import a GGUF #
Basic Import
wget https://huggingface.co/bartowski/Qwen2.5-14B-GGUF/resolve/main/Qwen2.5-14B-Q4_K_M.gguf
cat > Modelfile << 'EOF'
FROM ./Qwen2.5-14B-Q4_K_M.gguf
EOF
ollama create my-custom-model -f Modelfile
ollama run my-custom-model
Smart Import (with Optimized Settings)
cat > Modelfile << 'EOF'
FROM ./DeepSeek-R1-14B-Q4_K_M.gguf
PARAMETER num_ctx 32768
PARAMETER num_gpu_layers 999
PARAMETER num_thread 8
PARAMETER numa true
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.1
TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>
{{ end }}<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
SYSTEM """You are a helpful AI assistant."""
EOF
ollama create my-r1-custom -f Modelfile
ollama run my-r1-custom
Step 3: Modelfile Reference #
A Modelfile is like a Dockerfile for LLMs. Every line is an instruction.
Parameters Reference
| Parameter | What It Does | Default | Recommended Range |
|---|---|---|---|
temperature |
|||
| Creativity level | 0.8 | 0.2 (code) – 1.0 (creative) | |
top_p |
|||
| Nucleus sampling | 0.9 | 0.85 – 0.95 | |
top_k |
|||
| Top-K sampling | 40 | 20 – 100 | |
num_ctx |
|||
| Context window size | 2048 | 4096 – 65536 | |
num_gpu |
|||
| GPU layers | 0 (auto) | 999 (use all VRAM) | |
num_thread |
|||
| CPU threads | auto | 4 – 16 | |
repeat_penalty |
|||
| Penalize repetition | 1.1 | 1.0 – 1.2 | |
stop |
|||
| Stop sequences | varies | `< |
INSTRUCTION vs SYSTEM vs TEMPLATE
{% raw %}
SYSTEM """You are a helpful assistant."""
TEMPLATE """User: {{ .Prompt }}
Assistant: """
INSTRUCTION """Follow the user's instructions carefully."""
Three Production Configs
1. Coding Assistant
FROM qwen2.5:7b
PARAMETER temperature 0.2
PARAMETER top_p 0.85
PARAMETER num_ctx 65536
PARAMETER repeat_penalty 1.1
SYSTEM """You are an expert Python developer. Write clean, tested code."""
2. Creative Writer
FROM mistral
PARAMETER temperature 1.0
PARAMETER top_p 0.95
PARAMETER num_ctx 16384
SYSTEM """You are a novelist. Be vivid and descriptive."""
3. Customer Support
FROM llama4
PARAMETER temperature 0.5
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM """You are a helpful customer support agent.
Be polite, concise, and solution-oriented.
NEVER mention that you are an AI."""
Step 4: Advanced Techniques #
4.1 Multi-GPU Setup
FROM deepseek-r1:70b
PARAMETER num_gpu_layers 999
PARAMETER main_gpu 0
PARAMETER tensor_split "0.5,0.5"
4.2 LoRA Adapters (Experimental)
Some Ollama builds support LoRA adapters:
FROM base-model
ADAPTER ./my-finetune-lora.gguf
PARAMETER temperature 0.7
4.3 Custom Stop Tokens
DeepSeek-R1 and Qwen use different stop tokens:
TEMPLATE """<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""
PARAMETER stop "<|im_end|>"
PARAMETER stop "<|im_start|>"
TEMPLATE """User: {{ .Prompt }}
Assistant: """
PARAMETER stop "User:"
4.4 Emergency: VRAM Too Low
If you get "CUDA out of memory":
PARAMETER num_gpu_layers 24 # Only put 24 layers on GPU
PARAMETER num_thread 8 # Use 8 CPU threads for the rest
Step 5: GGUF from Ollama Models (Export) #
You can also export a model from Ollama back to a GGUF file:
ollama pull qwen2.5:7b
ollama export qwen2.5:7b ./my-export.gguf
./llama-cli -m ./my-export.gguf -p "Hello"
This is useful for:
- Moving models between machines without re-down
- Using the same model with multiple inference engines
- Sharing a specific quantization with teammates
Performance Cheat Sheet #
By GPU
| GPU | VRAM | Best GGUF Model | Expected Speed |
|---|---|---|---|
| RTX 3060 / 4060 | 12 GB | Qwen 2.5-14B (Q4_K_M) | 30-40 tok/s |
| RTX 4070 / 5070 | 12 GB | Qwen 2.5-14B (Q4_K_M) | 35-50 tok/s |
| RTX 4080 / 5080 | 16 GB | DeepSeek-R1-14B (Q4_K_M) | 30-45 tok/s |
| RTX 4090 / 5090 | 24 GB | DeepSeek-R1-32B (Q4_K_M) | 18-25 tok/s |
| Mac M2 Pro | 16 GB | Qwen 2.5-7B (Q4_K_M) | 15-25 tok/s |
| Mac M4 Max | 36 GB | Qwen 3.6-27B (Q4_K_M) | 20-30 tok/s |
CPU-Only Performance
| Model | Quant | RAM | Speed |
|---|---|---|---|
| Qwen 2.5-1.5B | Q4_K_M | 4 GB | 8-15 tok/s |
| Qwen 2.5-7B | Q4_K_M | 16 GB | 1-4 tok/s |
| Qwen 2.5-7B | Q2_K | 8 GB | 2-6 tok/s |
Common Pitfalls #
| Problem | Cause | Fix |
|---|---|---|
| "Model not found" after import | Modelfile path is wrong | Use absolute path: FROM /home/user/model.gguf |
| Gibberish output | Wrong chat template | The TEMPLATE line must match the model's expected format |
| Slow generation | Running on CPU | PARAMETER num_gpu_layers 999 |
| CUDA out of memory | Quantization too large for VRAM | Try smaller quant (Q3_K_M instead of Q4_K_M) |
| Import errors | Corrupt GGUF download | Re-download and verify checksum |
| Temperature not working | Set in Modelfile but overridden in API | Use the same temp in both places |
| Chinese text output | Wrong template or default system prompt | Add `PARAMETER stop "< |
The tl;dr #
Download:{% raw %}wget <huggingface-url>/Model-Q4_K_M.gguf
Create Modelfile:FROM ./Model.gguf
- your settings -
Import:
ollama create my-model -f Modelfile
Run:ollama run my-model
Profit: Free, private, local AI
Part of the Local LLM Guide — the definitive resource for running AI on your own hardware.