{"slug": "deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison", "title": "Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison", "summary": "A developer compared three local LLM inference engines — Ollama, llama.cpp, and vLLM — detailing their trade-offs in ease of use, quantization, batching, and hardware support. The writeup includes VRAM sizing rules of thumb (roughly params × 0.5 GB at Q4 to × 1.0 at Q8), a quantization quality/speed table, and setup commands for running 7B–70B models on consumer hardware with OpenAI-compatible APIs.", "body_md": "Run any model on your hardware — no API keys, no cloud, complete privacy.\n\nLocal LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: **Ollama** (easiest), **llama.cpp** (most flexible), **vLLM** (highest throughput).\n\n| Feature | Ollama | llama.cpp | vLLM | \n|---|---|---|---|\n| Ease of use | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | \n| Model formats | GGUF | GGUF | HF/Safetensors | \n| API compatible | ✅ OpenAI | ✅ OpenAI | ✅ OpenAI | \n| Multimodal | ✅ | ✅ | ✅ | \n| Batching | ❌ | ❌ | ✅ PagedAttention | \n| Quantization | Auto | Manual | Auto (AWQ/GPTQ) | \n| GPU/CPU | Both | Both | GPU (CUDA/ROCm) | \n| Best for | Devs, local chat | Embedded, edge | Production serving | \n\n| Model | Quantization | VRAM/RAM | CPU Only | \n|---|---|---|---|\n| 7B | Q4_K_M | 6 GB | ✅ Slow | \n| 7B | Q8_0 | 8 GB | ✅ Usable | \n| 13B | Q4_K_M | 10 GB | ✅ Slow | \n| 33B | Q4_K_M | 20 GB | ❌ | \n| 70B | Q4_K_M | 40 GB | ❌ | \n\n**Rule of thumb:** VRAM ≈ params × 0.5 (Q4) to 1.0 (Q8) in GB.\n\n```\n# Linux\ncurl -fsSL https://ollama.com/install.sh | sh\n\n# macOS\nbrew install ollama\n\n# Windows\nwinget install Ollama.Ollama\n# Pull and run (auto-downloads GGUF)\nollama run mistral:7b-instruct-q4_K_M\n\n# List local models\nollama list\n\n# Show model info\nollama show mistral:7b-instruct-q4_K_M\n# Start server (default port 11434)\nollama serve\n\n# Test API\ncurl -X POST http://localhost:11434/v1/chat/completions \\\n  -H \"Content-Type: application/json\" \\\n  -d '{\n    \"model\": \"mistral:7b-instruct-q4_K_M\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello!\"}],\n    \"stream\": false\n  }'\npython\nimport requests\n\ndef chat(messages, model=\"mistral:7b-instruct-q4_K_M\"):\n    response = requests.post(\n        \"http://localhost:11434/v1/chat/completions\",\n        json={\"model\": model, \"messages\": messages, \"stream\": False}\n    )\n    return response.json()[\"choices\"][0][\"message\"][\"content\"]\n\nprint(chat([{\"role\": \"user\", \"content\": \"Write a haiku about GPUs\"}]))\n# Modelfile\nFROM ./my-model.Q4_K_M.gguf\nTEMPLATE \"{{ .System }}\\n\\n{{ .Prompt }}\"\nPARAMETER temperature 0.7\nPARAMETER top_p 0.9\nPARAMETER num_ctx 8192\nSYSTEM \"You are a helpful coding assistant.\"\nollama create my-coder -f Modelfile\nollama run my-coder\ngit clone https://github.com/ggerganov/llama.cpp\ncd llama.cpp\nmake LLAMA_CUBLAS=1  # For NVIDIA GPU\n# or: make LLAMA_METAL=1  # For Apple Silicon\n# Download model (needs HF token for gated models)\nhuggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b\n\n# Convert\npython convert-hf-to-gguf.py ./llama-3.1-8b \\\n  --outfile ./llama-3.1-8b.Q4_K_M.gguf \\\n  --outtype q4_k_m\n```\n\n| Type | Size (7B) | Quality | Speed | \n|---|---|---|---|\n| Q2_K | ~2.8 GB | Low | Fastest | \n| Q3_K_M | ~3.3 GB | Medium | Fast | \n| **Q4_K_M** | **~3.8 GB** | **Good** | **Balanced** | \n| Q5_K_M | ~4.7 GB | Better | Slower | \n| Q6_K | ~5.3 GB | High | Slower | \n| Q8_0 | ~7.2 GB | Near fp16 | Slowest | \n\n```\n# Basic server\n./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080\n\n# With GPU offload (all layers)\n./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99\n\n# With flash attention (faster)\n./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa\npython\nfrom llama_cpp import Llama\n\nllm = Llama(\n    model_path=\"./model.Q4_K_M.gguf\",\n    n_ctx=8192,\n    n_gpu_layers=-1,  # -1 = all layers on GPU\n    verbose=False\n)\n\n# Chat completion\nresponse = llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    temperature=0.7,\n    max_tokens=512\n)\nprint(response[\"choices\"][0][\"message\"][\"content\"])\n\n# Streaming\nfor chunk in llm.create_chat_completion(\n    messages=[{\"role\": \"user\", \"content\": \"Write a story\"}],\n    stream=True\n):\n    delta = chunk[\"choices\"][0][\"delta\"]\n    if \"content\" in delta:\n        print(delta[\"content\"], end=\"\", flush=True)\n# Draft model (small) + target model (large)\n./llama-server -m ./llama-3.1-8b.Q4_K_M.gguf \\\n  -md ./llama-3.1-1b.Q4_K_M.gguf \\\n  -c 8192 --port 8080 -ngl 99\n# Requires CUDA 12.1+ and Python 3.10+\npip install vllm\n\n# For ROCm (AMD)\npip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0\n# HF model (auto-downloads)\nvllm serve meta-llama/Llama-3.1-8B-Instruct \\\n  --tensor-parallel-size 1 \\\n  --gpu-memory-utilization 0.9 \\\n  --max-model-len 8192 \\\n  --dtype auto \\\n  --api-key your-secret-key\n# AWQ (faster inference)\nvllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \\\n  --quantization awq \\\n  --gpu-memory-utilization 0.9\n\n# GPTQ\nvllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \\\n  --quantization gptq \\\n  --gpu-memory-utilization 0.9\n# 2 GPUs\nvllm serve meta-llama/Llama-3.1-70B-Instruct \\\n  --tensor-parallel-size 2 \\\n  --gpu-memory-utilization 0.9 \\\n  --max-model-len 4096\npython\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:8000/v1\",\n    api_key=\"your-secret-key\"\n)\n\n# Chat\nresponse = client.chat.completions.create(\n    model=\"meta-llama/Llama-3.1-8B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Hello!\"}],\n    temperature=0.7,\n    max_tokens=512\n)\nprint(response.choices[0].message.content)\n\n# Streaming\nstream = client.chat.completions.create(\n    model=\"meta-llama/Llama-3.1-8B-Instruct\",\n    messages=[{\"role\": \"user\", \"content\": \"Write a poem\"}],\n    stream=True\n)\nfor chunk in stream:\n    if chunk.choices[0].delta.content:\n        print(chunk.choices[0].delta.content, end=\"\", flush=True)\nModel: Llama-3.1-8B, Batch: 32, Context: 2048\n\nvLLM (PagedAttention):     ~450 tok/s\nllama.cpp (batch 32):      ~280 tok/s\nOllama (no batching):       ~120 tok/s\n```\n\n| Size | Model | Best For | \n|---|---|---|\n| 1B | SmolLM2-1.7B-Instruct | Edge, mobile | \n| 3B | Phi-3.5-mini-instruct | Fast, capable | \n| 7B | **Mistral-Nemo-12B-Instruct** | **Best 7B-class** | \n| 7B | Llama-3.1-8B-Instruct | Standard baseline | \n| 7B | Qwen2.5-7B-Instruct | Multilingual, code | \n| 13B | Nemotron-3-Ultra | Reasoning | \n\n| Size | Model | \n|---|---|\n| 7B | CodeQwen1.5-7B | \n| 7B | DeepSeek-Coder-V2-Lite | \n| 13B | CodeLlama-13B-Instruct | \n| 33B | DeepSeek-Coder-V2-33B | \n\n| Size | Model | \n|---|---|\n| 7B | Nemotron-3-Ultra | \n| 13B | QwQ-32B-Preview (distilled) | \n| 33B | Nemotron-3-Ultra-33B | \n\n```\n# Optimal thread count (physical cores)\n./llama-server -m model.gguf -t $(nproc --all) -ngl 99\n\n# Batch size for throughput\n./llama-server -m model.gguf -b 512 -ub 512\n\n# Flash attention (requires Hopper/Ada or Apple Silicon)\n./llama-server -m model.gguf -fa\n# Increase batch size\nvllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256\n\n# Chunked prefill (long contexts)\nvllm serve model --enable-chunked-prefill --max-model-len 32768\n\n# Prefix caching (shared prompts)\nvllm serve model --enable-prefix-caching\n# Environment variables\nOLLAMA_NUM_PARALLEL=4       # Concurrent requests\nOLLAMA_MAX_LOADED_MODELS=2  # Keep models in VRAM\nOLLAMA_FLASH_ATTENTION=1    # Enable flash attention\n```\n\n| Setup | Hardware | Models | Est. Cost | \n|---|---|---|---|\n| **Ollama + RTX 3090** | $800 used | 7B-33B Q4 | **$15/mo** electricity | \n| **llama.cpp + RTX 4090** | $1600 | 7B-70B Q4 | **$25/mo** electricity | \n| **vLLM + 2×A100 80GB** | $30k | 70B fp8 | **$500/mo** colocation | \n| **Cloud API (OpenRouter)** | — | Any | **$50-500/mo** usage | \n\n**Break-even:** Local GPU pays for itself in 6-18 months vs cloud APIs.\n\n| Problem | Fix | \n|---|---|\n| OOM on GPU | Reduce `n_gpu_layers` , use smaller quantization, offload to CPU | \n| Slow CPU inference | Use `Q4_K_M` , enable BLAS (`LLAMA_BLAS=1` ), reduce context | \n| Model not loading | Check GGUF compatibility, try `llama.cpp` convert script | \n| API connection refused | Check port, firewall, `OLLAMA_HOST=0.0.0.0` | \n| Poor quality | Increase temperature, try larger model, better quantization | \n\n```\nSTART: What's your use case?\n│\n├─→ \"Just chat / quick tests\" → **Ollama**\n│\n├─→ \"Embed in app / edge device / custom quant\" → **llama.cpp**\n│\n├─→ \"Serve many users / high throughput / production\" → **vLLM**\n│\n└─→ \"Apple Silicon Mac\" → **Ollama** or **llama.cpp (Metal)**\n```\n\nLocal LLMs are production-ready. Choose:\n\nAll three expose **OpenAI-compatible APIs** — swap providers without code changes.\n\n**Next:** Add RAG (see previous tutorial), function calling, or fine-tune your own LoRA.\n\n*Part of the Local LLM series on Dev.to*", "url": "https://wpnews.pro/news/deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison", "canonical_source": "https://dev.to/ipsphil_digital/deploying-llms-locally-ollama-llamacpp-and-vllm-comparison-1bln", "published_at": "2026-10-12 16:37:34+00:00", "updated_at": "2026-10-12 16:50:18.866739+00:00", "lang": "en", "topics": ["large-language-models", "ai-infrastructure", "ai-tools", "mlops", "developer-tools"], "entities": ["Ollama", "llama.cpp", "vLLM", "GGUF", "PagedAttention", "Meta", "Llama 3.1", "Mistral"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison", "markdown": "https://wpnews.pro/news/deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison.md", "text": "https://wpnews.pro/news/deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison.txt", "jsonld": "https://wpnews.pro/news/deploying-llms-locally-ollama-llama-cpp-and-vllm-comparison.jsonld"}}