# Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

> Source: <https://dev.to/ipsphil_digital/deploying-llms-locally-ollama-llamacpp-and-vllm-comparison-1bln>
> Published: 2026-10-12 16:37:34+00:00

Run any model on your hardware — no API keys, no cloud, complete privacy.

Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: **Ollama** (easiest), **llama.cpp** (most flexible), **vLLM** (highest throughput).

| Feature | Ollama | llama.cpp | vLLM | 
|---|---|---|---|
| Ease of use | ⭐⭐⭐⭐⭐ | ⭐⭐⭐ | ⭐⭐ | 
| Model formats | GGUF | GGUF | HF/Safetensors | 
| API compatible | ✅ OpenAI | ✅ OpenAI | ✅ OpenAI | 
| Multimodal | ✅ | ✅ | ✅ | 
| Batching | ❌ | ❌ | ✅ PagedAttention | 
| Quantization | Auto | Manual | Auto (AWQ/GPTQ) | 
| GPU/CPU | Both | Both | GPU (CUDA/ROCm) | 
| Best for | Devs, local chat | Embedded, edge | Production serving | 

| Model | Quantization | VRAM/RAM | CPU Only | 
|---|---|---|---|
| 7B | Q4_K_M | 6 GB | ✅ Slow | 
| 7B | Q8_0 | 8 GB | ✅ Usable | 
| 13B | Q4_K_M | 10 GB | ✅ Slow | 
| 33B | Q4_K_M | 20 GB | ❌ | 
| 70B | Q4_K_M | 40 GB | ❌ | 

**Rule of thumb:** VRAM ≈ params × 0.5 (Q4) to 1.0 (Q8) in GB.

```
# Linux
curl -fsSL https://ollama.com/install.sh | sh

# macOS
brew install ollama

# Windows
winget install Ollama.Ollama
# Pull and run (auto-downloads GGUF)
ollama run mistral:7b-instruct-q4_K_M

# List local models
ollama list

# Show model info
ollama show mistral:7b-instruct-q4_K_M
# Start server (default port 11434)
ollama serve

# Test API
curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral:7b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'
python
import requests

def chat(messages, model="mistral:7b-instruct-q4_K_M"):
    response = requests.post(
        "http://localhost:11434/v1/chat/completions",
        json={"model": model, "messages": messages, "stream": False}
    )
    return response.json()["choices"][0]["message"]["content"]

print(chat([{"role": "user", "content": "Write a haiku about GPUs"}]))
# Modelfile
FROM ./my-model.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n\n{{ .Prompt }}"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM "You are a helpful coding assistant."
ollama create my-coder -f Modelfile
ollama run my-coder
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1  # For NVIDIA GPU
# or: make LLAMA_METAL=1  # For Apple Silicon
# Download model (needs HF token for gated models)
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b

# Convert
python convert-hf-to-gguf.py ./llama-3.1-8b \
  --outfile ./llama-3.1-8b.Q4_K_M.gguf \
  --outtype q4_k_m
```

| Type | Size (7B) | Quality | Speed | 
|---|---|---|---|
| Q2_K | ~2.8 GB | Low | Fastest | 
| Q3_K_M | ~3.3 GB | Medium | Fast | 
| **Q4_K_M** | **~3.8 GB** | **Good** | **Balanced** | 
| Q5_K_M | ~4.7 GB | Better | Slower | 
| Q6_K | ~5.3 GB | High | Slower | 
| Q8_0 | ~7.2 GB | Near fp16 | Slowest | 

```
# Basic server
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080

# With GPU offload (all layers)
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99

# With flash attention (faster)
./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa
python
from llama_cpp import Llama

llm = Llama(
    model_path="./model.Q4_K_M.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,  # -1 = all layers on GPU
    verbose=False
)

# Chat completion
response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response["choices"][0]["message"]["content"])

# Streaming
for chunk in llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
):
    delta = chunk["choices"][0]["delta"]
    if "content" in delta:
        print(delta["content"], end="", flush=True)
# Draft model (small) + target model (large)
./llama-server -m ./llama-3.1-8b.Q4_K_M.gguf \
  -md ./llama-3.1-1b.Q4_K_M.gguf \
  -c 8192 --port 8080 -ngl 99
# Requires CUDA 12.1+ and Python 3.10+
pip install vllm

# For ROCm (AMD)
pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0
# HF model (auto-downloads)
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --dtype auto \
  --api-key your-secret-key
# AWQ (faster inference)
vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.9

# GPTQ
vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \
  --quantization gptq \
  --gpu-memory-utilization 0.9
# 2 GPUs
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 4096
python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key"
)

# Chat
response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a poem"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
Model: Llama-3.1-8B, Batch: 32, Context: 2048

vLLM (PagedAttention):     ~450 tok/s
llama.cpp (batch 32):      ~280 tok/s
Ollama (no batching):       ~120 tok/s
```

| Size | Model | Best For | 
|---|---|---|
| 1B | SmolLM2-1.7B-Instruct | Edge, mobile | 
| 3B | Phi-3.5-mini-instruct | Fast, capable | 
| 7B | **Mistral-Nemo-12B-Instruct** | **Best 7B-class** | 
| 7B | Llama-3.1-8B-Instruct | Standard baseline | 
| 7B | Qwen2.5-7B-Instruct | Multilingual, code | 
| 13B | Nemotron-3-Ultra | Reasoning | 

| Size | Model | 
|---|---|
| 7B | CodeQwen1.5-7B | 
| 7B | DeepSeek-Coder-V2-Lite | 
| 13B | CodeLlama-13B-Instruct | 
| 33B | DeepSeek-Coder-V2-33B | 

| Size | Model | 
|---|---|
| 7B | Nemotron-3-Ultra | 
| 13B | QwQ-32B-Preview (distilled) | 
| 33B | Nemotron-3-Ultra-33B | 

```
# Optimal thread count (physical cores)
./llama-server -m model.gguf -t $(nproc --all) -ngl 99

# Batch size for throughput
./llama-server -m model.gguf -b 512 -ub 512

# Flash attention (requires Hopper/Ada or Apple Silicon)
./llama-server -m model.gguf -fa
# Increase batch size
vllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256

# Chunked prefill (long contexts)
vllm serve model --enable-chunked-prefill --max-model-len 32768

# Prefix caching (shared prompts)
vllm serve model --enable-prefix-caching
# Environment variables
OLLAMA_NUM_PARALLEL=4       # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2  # Keep models in VRAM
OLLAMA_FLASH_ATTENTION=1    # Enable flash attention
```

| Setup | Hardware | Models | Est. Cost | 
|---|---|---|---|
| **Ollama + RTX 3090** | $800 used | 7B-33B Q4 | **$15/mo** electricity | 
| **llama.cpp + RTX 4090** | $1600 | 7B-70B Q4 | **$25/mo** electricity | 
| **vLLM + 2×A100 80GB** | $30k | 70B fp8 | **$500/mo** colocation | 
| **Cloud API (OpenRouter)** | — | Any | **$50-500/mo** usage | 

**Break-even:** Local GPU pays for itself in 6-18 months vs cloud APIs.

| Problem | Fix | 
|---|---|
| OOM on GPU | Reduce `n_gpu_layers` , use smaller quantization, offload to CPU | 
| Slow CPU inference | Use `Q4_K_M` , enable BLAS (`LLAMA_BLAS=1` ), reduce context | 
| Model not loading | Check GGUF compatibility, try `llama.cpp` convert script | 
| API connection refused | Check port, firewall, `OLLAMA_HOST=0.0.0.0` | 
| Poor quality | Increase temperature, try larger model, better quantization | 

```
START: What's your use case?
│
├─→ "Just chat / quick tests" → **Ollama**
│
├─→ "Embed in app / edge device / custom quant" → **llama.cpp**
│
├─→ "Serve many users / high throughput / production" → **vLLM**
│
└─→ "Apple Silicon Mac" → **Ollama** or **llama.cpp (Metal)**
```

Local LLMs are production-ready. Choose:

All three expose **OpenAI-compatible APIs** — swap providers without code changes.

**Next:** Add RAG (see previous tutorial), function calling, or fine-tune your own LoRA.

*Part of the Local LLM series on Dev.to*