Originally published on kuryzhev.cloud
We thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here's what actually works — and what we'd do differently from day one.
Our homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else's Terraform. All of that involves a lot of back-and-forth with language models.
The problem with cloud-based LLM APIs — ChatGPT, Claude, Gemini — is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can't reach the internet, and we wanted consistent tooling across both environments.
We chose Ollama for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters — it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using curl
, custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not.
The first host we tried had an NVIDIA GTX 1060 3GB installed. We ran ollama run llama3
, watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU.
We burned two hours checking the wrong things — model quantization, RAM speed, thermal throttling — before running ollama ps
and seeing 100% CPU
next to the loaded model. Then we checked the logs with OLLAMA_DEBUG=1 ollama serve
and found the real message:
msg="no GPU detected"
The GTX 1060 was visible to nvidia-smi
. The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client's perspective — Ollama simply fell back to CPU without surfacing any error to the caller.
Watch out for this: nvidia-smi
showing your GPU does not mean Ollama will use it. Always verify with OLLAMA_DEBUG=1
and look for offload layers: 32/32
in the output. That line confirms full GPU acceleration. Anything less means partial or zero offload.
The fix required three steps: updating the CUDA toolkit to 11.8, pinning nvidia-container-toolkit
to version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring /etc/docker/daemon.json
had "default-runtime": "nvidia"
set. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience.
We also verified the CUDA version properly this time:
nvcc --version
docker info | grep -i runtime
OLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i "offload layers"
Once GPU acceleration was working, we got greedy and pulled a larger model. codellama:13b
seemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways.
The OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn't pushed yet. The codellama:13b
model in Q4_K_M quantization needs roughly 8.5GB of VRAM — and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn't.
The root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, OLLAMA_MAX_LOADED_MODELS
controls how many models stay resident in memory simultaneously. In earlier versions, there's no eviction control at all. We hadn't set either OLLAMA_MAX_LOADED_MODELS
or OLLAMA_NUM_PARALLEL
, so Ollama defaulted to greedy behavior — whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure.
Watch out for this: OLLAMA_NUM_PARALLEL
defaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware.
The correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set mem_limit: 10g
, and added environment variables to enforce single-model, single-request behavior.
This one is embarrassing to admit. The default ollama serve
binds to 127.0.0.1
, which is fine for bare-metal installs. But our Docker Compose setup used ports: "11434:11434"
without thinking through what that means on a flat home network. It published the port to every interface on the host — which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we'd misconfigured months earlier.
Ollama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call GET /api/tags
to enumerate loaded models, send inference requests, or — and this is the one that actually worried us — trigger a POST /api/pull
to download a multi-gigabyte model and exhaust disk space. There's no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume.
We also had OLLAMA_HOST=0.0.0.0
set inside the container, which is necessary for the container's internal binding but easy to misread as "only affects the container namespace." It doesn't change the fact that the Docker port mapping exposes it externally.
The fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy's published port to 127.0.0.1:8080
on the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt location /api/tags
block in Nginx.
The current setup avoids all three mistakes. Here's the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the nvidia
runtime, hard memory caps, and an Nginx auth proxy — all wired together with a named volume for model persistence.
This Docker Compose file defines the full hardened stack. Read the inline comments carefully — several of the environment variables are non-obvious in their effect:
version: "3.9"
services:
ollama:
image: ollama/ollama:0.1.38 # pinned — avoid surprise API changes
container_name: ollama
restart: unless-stopped
runtime: nvidia # requires nvidia-container-toolkit
environment:
- OLLAMA_HOST=0.0.0.0 # bind inside container; proxy handles external auth
- OLLAMA_NUM_PARALLEL=1 # prevent concurrent request memory spikes
- OLLAMA_MAX_LOADED_MODELS=1 # evict previous model before new one
- OLLAMA_DEBUG=0 # set to 1 temporarily to verify GPU offload layers
volumes:
- ollama_models:/root/.ollama # persist models across container rebuilds
networks:
- ollama_internal # isolated from critical services network
deploy:
resources:
limits:
memory: 10g # hard cap — prevents OOM-killing neighbors
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"]
interval: 30s
timeout: 10s
retries: 3
start_period: 20s # model server takes ~15s to initialize
ollama-proxy:
image: nginx:1.25-alpine
container_name: ollama-proxy
restart: unless-stopped
ports:
- "127.0.0.1:8080:80" # bind to loopback only; use Tailscale for remote access
volumes:
- ./nginx/ollama.conf:/etc/nginx/conf.d/default.conf:ro
- ./nginx/.htpasswd:/etc/nginx/.htpasswd:ro
networks:
- ollama_internal
depends_on:
ollama:
condition: service_healthy
volumes:
ollama_models:
driver: local
networks:
ollama_internal:
driver: bridge
internal: false # set true if Ollama should have no outbound internet
The Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting proxy_buffering off
(which causes the response to appear all at once instead of streaming) and leaving proxy_read_timeout
at the default 60 seconds, which causes 504 errors mid-generation on longer prompts:
server {
listen 80;
server_name _;
auth_basic "Ollama API";
auth_basic_user_file /etc/nginx/.htpasswd;
location / {
proxy_pass http://ollama:11434;
proxy_http_version 1.1;
proxy_set_header Connection '';
proxy_buffering off;
proxy_cache off;
proxy_read_timeout 300s;
proxy_connect_timeout 10s;
proxy_send_timeout 300s;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
}
location /api/tags {
auth_basic off;
proxy_pass http://ollama:11434/api/tags;
}
}
For model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: codellama:7b
in Q4_K_M for quick completions and inline code suggestions, and mistral:7b-instruct-v0.2-q4_K_M
for longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they're genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM.
We also pin models using digest references in our Modelfiles — FROM llama3:8b-instruct@sha256:abc123...
— so a weekly ollama pull
cron job doesn't silently swap out a model we've tuned prompts for. Model versioning matters more than most people realize when you're building automation on top of LLM output.
For observability, Ollama has no native Prometheus endpoint, so we rely on ollama ps
and ollama list
in a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It's not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a state. More details on our homelab monitoring approach are over at kuryzhev.cloud.
Running a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first — hopefully this saves you the OOM crashes.