Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

A developer compared three local LLM inference engines — Ollama, llama.cpp, and vLLM — detailing their trade-offs in ease of use, quantization, batching, and hardware support. The writeup includes VRAM sizing rules of thumb (roughly params × 0.5 GB at Q4 to × 1.0 at Q8), a quantization quality/speed table, and setup commands for running 7B–70B models on consumer hardware with OpenAI-compatible APIs.

Run any model on your hardware — no API keys, no cloud, complete privacy. Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: Ollama easiest , llama.cpp most flexible , vLLM highest throughput . | Feature | Ollama | llama.cpp | vLLM | |---|---|---|---| | Ease of use | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | | Model formats | GGUF | GGUF | HF/Safetensors | | API compatible | ✅ OpenAI | ✅ OpenAI | ✅ OpenAI | | Multimodal | ✅ | ✅ | ✅ | | Batching | ❌ | ❌ | ✅ PagedAttention | | Quantization | Auto | Manual | Auto AWQ/GPTQ | | GPU/CPU | Both | Both | GPU CUDA/ROCm | | Best for | Devs, local chat | Embedded, edge | Production serving | | Model | Quantization | VRAM/RAM | CPU Only | |---|---|---|---| | 7B | Q4 K M | 6 GB | ✅ Slow | | 7B | Q8 0 | 8 GB | ✅ Usable | | 13B | Q4 K M | 10 GB | ✅ Slow | | 33B | Q4 K M | 20 GB | ❌ | | 70B | Q4 K M | 40 GB | ❌ | Rule of thumb: VRAM ≈ params × 0.5 Q4 to 1.0 Q8 in GB. Linux curl -fsSL https://ollama.com/install.sh | sh macOS brew install ollama Windows winget install Ollama.Ollama Pull and run auto-downloads GGUF ollama run mistral:7b-instruct-q4 K M List local models ollama list Show model info ollama show mistral:7b-instruct-q4 K M Start server default port 11434 ollama serve Test API curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "mistral:7b-instruct-q4 K M", "messages": {"role": "user", "content": "Hello "} , "stream": false }' python import requests def chat messages, model="mistral:7b-instruct-q4 K M" : response = requests.post "http://localhost:11434/v1/chat/completions", json={"model": model, "messages": messages, "stream": False} return response.json "choices" 0 "message" "content" print chat {"role": "user", "content": "Write a haiku about GPUs"} Modelfile FROM ./my-model.Q4 K M.gguf TEMPLATE "{{ .System }}\n\n{{ .Prompt }}" PARAMETER temperature 0.7 PARAMETER top p 0.9 PARAMETER num ctx 8192 SYSTEM "You are a helpful coding assistant." ollama create my-coder -f Modelfile ollama run my-coder git clone https://github.com/ggerganov/llama.cpp cd llama.cpp make LLAMA CUBLAS=1 For NVIDIA GPU or: make LLAMA METAL=1 For Apple Silicon Download model needs HF token for gated models huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b Convert python convert-hf-to-gguf.py ./llama-3.1-8b \ --outfile ./llama-3.1-8b.Q4 K M.gguf \ --outtype q4 k m | Type | Size 7B | Quality | Speed | |---|---|---|---| | Q2 K | ~2.8 GB | Low | Fastest | | Q3 K M | ~3.3 GB | Medium | Fast | | Q4 K M | ~3.8 GB | Good | Balanced | | Q5 K M | ~4.7 GB | Better | Slower | | Q6 K | ~5.3 GB | High | Slower | | Q8 0 | ~7.2 GB | Near fp16 | Slowest | Basic server ./llama-server -m ./model.Q4 K M.gguf -c 8192 --port 8080 With GPU offload all layers ./llama-server -m ./model.Q4 K M.gguf -c 8192 --port 8080 -ngl 99 With flash attention faster ./llama-server -m ./model.Q4 K M.gguf -c 8192 --port 8080 -ngl 99 -fa python from llama cpp import Llama llm = Llama model path="./model.Q4 K M.gguf", n ctx=8192, n gpu layers=-1, -1 = all layers on GPU verbose=False Chat completion response = llm.create chat completion messages= {"role": "user", "content": "Hello "} , temperature=0.7, max tokens=512 print response "choices" 0 "message" "content" Streaming for chunk in llm.create chat completion messages= {"role": "user", "content": "Write a story"} , stream=True : delta = chunk "choices" 0 "delta" if "content" in delta: print delta "content" , end="", flush=True Draft model small + target model large ./llama-server -m ./llama-3.1-8b.Q4 K M.gguf \ -md ./llama-3.1-1b.Q4 K M.gguf \ -c 8192 --port 8080 -ngl 99 Requires CUDA 12.1+ and Python 3.10+ pip install vllm For ROCm AMD pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0 HF model auto-downloads vllm serve meta-llama/Llama-3.1-8B-Instruct \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.9 \ --max-model-len 8192 \ --dtype auto \ --api-key your-secret-key AWQ faster inference vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \ --quantization awq \ --gpu-memory-utilization 0.9 GPTQ vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \ --quantization gptq \ --gpu-memory-utilization 0.9 2 GPUs vllm serve meta-llama/Llama-3.1-70B-Instruct \ --tensor-parallel-size 2 \ --gpu-memory-utilization 0.9 \ --max-model-len 4096 python from openai import OpenAI client = OpenAI base url="http://localhost:8000/v1", api key="your-secret-key" Chat response = client.chat.completions.create model="meta-llama/Llama-3.1-8B-Instruct", messages= {"role": "user", "content": "Hello "} , temperature=0.7, max tokens=512 print response.choices 0 .message.content Streaming stream = client.chat.completions.create model="meta-llama/Llama-3.1-8B-Instruct", messages= {"role": "user", "content": "Write a poem"} , stream=True for chunk in stream: if chunk.choices 0 .delta.content: print chunk.choices 0 .delta.content, end="", flush=True Model: Llama-3.1-8B, Batch: 32, Context: 2048 vLLM PagedAttention : ~450 tok/s llama.cpp batch 32 : ~280 tok/s Ollama no batching : ~120 tok/s | Size | Model | Best For | |---|---|---| | 1B | SmolLM2-1.7B-Instruct | Edge, mobile | | 3B | Phi-3.5-mini-instruct | Fast, capable | | 7B | Mistral-Nemo-12B-Instruct | Best 7B-class | | 7B | Llama-3.1-8B-Instruct | Standard baseline | | 7B | Qwen2.5-7B-Instruct | Multilingual, code | | 13B | Nemotron-3-Ultra | Reasoning | | Size | Model | |---|---| | 7B | CodeQwen1.5-7B | | 7B | DeepSeek-Coder-V2-Lite | | 13B | CodeLlama-13B-Instruct | | 33B | DeepSeek-Coder-V2-33B | | Size | Model | |---|---| | 7B | Nemotron-3-Ultra | | 13B | QwQ-32B-Preview distilled | | 33B | Nemotron-3-Ultra-33B | Optimal thread count physical cores ./llama-server -m model.gguf -t $ nproc --all -ngl 99 Batch size for throughput ./llama-server -m model.gguf -b 512 -ub 512 Flash attention requires Hopper/Ada or Apple Silicon ./llama-server -m model.gguf -fa Increase batch size vllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256 Chunked prefill long contexts vllm serve model --enable-chunked-prefill --max-model-len 32768 Prefix caching shared prompts vllm serve model --enable-prefix-caching Environment variables OLLAMA NUM PARALLEL=4 Concurrent requests OLLAMA MAX LOADED MODELS=2 Keep models in VRAM OLLAMA FLASH ATTENTION=1 Enable flash attention | Setup | Hardware | Models | Est. Cost | |---|---|---|---| | Ollama + RTX 3090 | $800 used | 7B-33B Q4 | $15/mo electricity | | llama.cpp + RTX 4090 | $1600 | 7B-70B Q4 | $25/mo electricity | | vLLM + 2×A100 80GB | $30k | 70B fp8 | $500/mo colocation | | Cloud API OpenRouter | — | Any | $50-500/mo usage | Break-even: Local GPU pays for itself in 6-18 months vs cloud APIs. | Problem | Fix | |---|---| | OOM on GPU | Reduce n gpu layers , use smaller quantization, offload to CPU | | Slow CPU inference | Use Q4 K M , enable BLAS LLAMA BLAS=1 , reduce context | | Model not loading | Check GGUF compatibility, try llama.cpp convert script | | API connection refused | Check port, firewall, OLLAMA HOST=0.0.0.0 | | Poor quality | Increase temperature, try larger model, better quantization | START: What's your use case? │ ├─→ "Just chat / quick tests" → Ollama │ ├─→ "Embed in app / edge device / custom quant" → llama.cpp │ ├─→ "Serve many users / high throughput / production" → vLLM │ └─→ "Apple Silicon Mac" → Ollama or llama.cpp Metal Local LLMs are production-ready. Choose: All three expose OpenAI-compatible APIs — swap providers without code changes. Next: Add RAG see previous tutorial , function calling, or fine-tune your own LoRA. Part of the Local LLM series on Dev.to