Deploying LLMs Locally: Ollama, llama.cpp, and vLLM Comparison

Run any model on your hardware — no API keys, no cloud, complete privacy.

Local LLMs have matured. You can now run 7B-70B models on consumer hardware with near-cloud performance. This guide covers the three main engines: Ollama (easiest), llama.cpp (most flexible), vLLM (highest throughput).

Rule of thumb: VRAM ≈ params × 0.5 (Q4) to 1.0 (Q8) in GB.

curl -fsSL https://ollama.com/install.sh | sh

brew install ollama

winget install Ollama.Ollama
ollama run mistral:7b-instruct-q4_K_M

ollama list

ollama show mistral:7b-instruct-q4_K_M
ollama serve

curl -X POST http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "mistral:7b-instruct-q4_K_M",
    "messages": [{"role": "user", "content": "Hello!"}],
    "stream": false
  }'
python
import requests

def chat(messages, model="mistral:7b-instruct-q4_K_M"):
    response = requests.post(
        "http://localhost:11434/v1/chat/completions",
        json={"model": model, "messages": messages, "stream": False}
    )
    return response.json()["choices"][0]["message"]["content"]

print(chat([{"role": "user", "content": "Write a haiku about GPUs"}]))
FROM ./my-model.Q4_K_M.gguf
TEMPLATE "{{ .System }}\n\n{{ .Prompt }}"
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
SYSTEM "You are a helpful coding assistant."
ollama create my-coder -f Modelfile
ollama run my-coder
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make LLAMA_CUBLAS=1  # For NVIDIA GPU
huggingface-cli download meta-llama/Llama-3.1-8B-Instruct --local-dir ./llama-3.1-8b

python convert-hf-to-gguf.py ./llama-3.1-8b \
  --outfile ./llama-3.1-8b.Q4_K_M.gguf \
  --outtype q4_k_m

./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080

./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99

./llama-server -m ./model.Q4_K_M.gguf -c 8192 --port 8080 -ngl 99 -fa
python
from llama_cpp import Llama

llm = Llama(
    model_path="./model.Q4_K_M.gguf",
    n_ctx=8192,
    n_gpu_layers=-1,  # -1 = all layers on GPU
    verbose=False
)

response = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response["choices"][0]["message"]["content"])

for chunk in llm.create_chat_completion(
    messages=[{"role": "user", "content": "Write a story"}],
    stream=True
):
    delta = chunk["choices"][0]["delta"]
    if "content" in delta:
        print(delta["content"], end="", flush=True)
./llama-server -m ./llama-3.1-8b.Q4_K_M.gguf \
  -md ./llama-3.1-1b.Q4_K_M.gguf \
  -c 8192 --port 8080 -ngl 99
pip install vllm

pip install vllm --extra-index-url https://download.pytorch.org/whl/rocm6.0
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 8192 \
  --dtype auto \
  --api-key your-secret-key
vllm serve TheBloke/Llama-3.1-8B-Instruct-AWQ \
  --quantization awq \
  --gpu-memory-utilization 0.9

vllm serve TheBloke/Llama-3.1-8B-Instruct-GPTQ \
  --quantization gptq \
  --gpu-memory-utilization 0.9
vllm serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.9 \
  --max-model-len 4096
python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key"
)

response = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    temperature=0.7,
    max_tokens=512
)
print(response.choices[0].message.content)

stream = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Write a poem"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
Model: Llama-3.1-8B, Batch: 32, Context: 2048

vLLM (PagedAttention):     ~450 tok/s
llama.cpp (batch 32):      ~280 tok/s
Ollama (no batching):       ~120 tok/s

./llama-server -m model.gguf -t $(nproc --all) -ngl 99

./llama-server -m model.gguf -b 512 -ub 512

./llama-server -m model.gguf -fa
vllm serve model --max-num-batched-tokens 8192 --max-num-seqs 256

vllm serve model --enable-chunked-prefill --max-model-len 32768

vllm serve model --enable-prefix-caching
OLLAMA_NUM_PARALLEL=4       # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2  # Keep models in VRAM
OLLAMA_FLASH_ATTENTION=1    # Enable flash attention

Break-even: Local GPU pays for itself in 6-18 months vs cloud APIs.

START: What's your use case?
│
├─→ "Just chat / quick tests" → **Ollama**
│
├─→ "Embed in app / edge device / custom quant" → **llama.cpp**
│
├─→ "Serve many users / high throughput / production" → **vLLM**
│
└─→ "Apple Silicon Mac" → **Ollama** or **llama.cpp (Metal)**

Local LLMs are production-ready. Choose:

All three expose OpenAI-compatible APIs — swap providers without code changes.

Next: Add RAG (see previous tutorial), function calling, or fine-tune your own LoRA.

Part of the Local LLM series on Dev.to