# Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs > Source: > Published: 2026-06-26 07:02:10+00:00 *Originally published on kuryzhev.cloud* We thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here's what actually works — and what we'd do differently from day one. Our homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else's Terraform. All of that involves a lot of back-and-forth with language models. The problem with cloud-based LLM APIs — ChatGPT, Claude, Gemini — is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can't reach the internet, and we wanted consistent tooling across both environments. We chose [Ollama](https://ollama.com/) for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters — it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using `curl` , custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not. The first host we tried had an NVIDIA GTX 1060 3GB installed. We ran `ollama run llama3` , watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU. We burned two hours checking the wrong things — model quantization, RAM speed, thermal throttling — before running `ollama ps` and seeing `100% CPU` next to the loaded model. Then we checked the logs with `OLLAMA_DEBUG=1 ollama serve` and found the real message: ``` msg="no GPU detected" ``` The GTX 1060 was visible to `nvidia-smi` . The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client's perspective — Ollama simply fell back to CPU without surfacing any error to the caller. **Watch out for this:** `nvidia-smi` showing your GPU does not mean Ollama will use it. Always verify with `OLLAMA_DEBUG=1` and look for `offload layers: 32/32` in the output. That line confirms full GPU acceleration. Anything less means partial or zero offload. The fix required three steps: updating the CUDA toolkit to 11.8, pinning `nvidia-container-toolkit` to version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring `/etc/docker/daemon.json` had `"default-runtime": "nvidia"` set. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience. We also verified the CUDA version properly this time: ``` # Check CUDA version — must be >= 11.8 for Ollama GPU support nvcc --version # Confirm Docker is using the nvidia runtime docker info | grep -i runtime # Verify GPU offload after starting Ollama with debug logging OLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i "offload layers" ``` Once GPU acceleration was working, we got greedy and pulled a larger model. `codellama:13b` seemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways. The OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn't pushed yet. The `codellama:13b` model in Q4_K_M quantization needs roughly 8.5GB of VRAM — and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn't. The root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, `OLLAMA_MAX_LOADED_MODELS` controls how many models stay resident in memory simultaneously. In earlier versions, there's no eviction control at all. We hadn't set either `OLLAMA_MAX_LOADED_MODELS` or `OLLAMA_NUM_PARALLEL` , so Ollama defaulted to greedy behavior — loading whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure. **Watch out for this:** `OLLAMA_NUM_PARALLEL` defaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware. The correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set `mem_limit: 10g` , and added environment variables to enforce single-model, single-request behavior. This one is embarrassing to admit. The default `ollama serve` binds to `127.0.0.1` , which is fine for bare-metal installs. But our Docker Compose setup used `ports: "11434:11434"` without thinking through what that means on a flat home network. It published the port to every interface on the host — which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we'd misconfigured months earlier. Ollama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call `GET /api/tags` to enumerate loaded models, send inference requests, or — and this is the one that actually worried us — trigger a `POST /api/pull` to download a multi-gigabyte model and exhaust disk space. There's no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume. We also had `OLLAMA_HOST=0.0.0.0` set inside the container, which is necessary for the container's internal binding but easy to misread as "only affects the container namespace." It doesn't change the fact that the Docker port mapping exposes it externally. The fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy's published port to `127.0.0.1:8080` on the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt `location /api/tags` block in Nginx. The current setup avoids all three mistakes. Here's the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the `nvidia` runtime, hard memory caps, and an Nginx auth proxy — all wired together with a named volume for model persistence. This Docker Compose file defines the full hardened stack. Read the inline comments carefully — several of the environment variables are non-obvious in their effect: ``` # docker-compose.yml — Production-hardened Ollama stack for homelab # Tested with Ollama v0.1.38, Docker 24.x, nvidia-container-toolkit 1.14.3 version: "3.9" services: ollama: image: ollama/ollama:0.1.38 # pinned — avoid surprise API changes container_name: ollama restart: unless-stopped runtime: nvidia # requires nvidia-container-toolkit environment: - OLLAMA_HOST=0.0.0.0 # bind inside container; proxy handles external auth - OLLAMA_NUM_PARALLEL=1 # prevent concurrent request memory spikes - OLLAMA_MAX_LOADED_MODELS=1 # evict previous model before loading new one - OLLAMA_DEBUG=0 # set to 1 temporarily to verify GPU offload layers volumes: - ollama_models:/root/.ollama # persist models across container rebuilds networks: - ollama_internal # isolated from critical services network deploy: resources: limits: memory: 10g # hard cap — prevents OOM-killing neighbors reservations: devices: - driver: nvidia count: 1 capabilities: [gpu] healthcheck: test: ["CMD", "curl", "-f", "http://localhost:11434/api/tags"] interval: 30s timeout: 10s retries: 3 start_period: 20s # model server takes ~15s to initialize ollama-proxy: image: nginx:1.25-alpine container_name: ollama-proxy restart: unless-stopped ports: - "127.0.0.1:8080:80" # bind to loopback only; use Tailscale for remote access volumes: - ./nginx/ollama.conf:/etc/nginx/conf.d/default.conf:ro - ./nginx/.htpasswd:/etc/nginx/.htpasswd:ro networks: - ollama_internal depends_on: ollama: condition: service_healthy volumes: ollama_models: driver: local networks: ollama_internal: driver: bridge internal: false # set true if Ollama should have no outbound internet ``` The Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting `proxy_buffering off` (which causes the response to appear all at once instead of streaming) and leaving `proxy_read_timeout` at the default 60 seconds, which causes 504 errors mid-generation on longer prompts: ``` # nginx/ollama.conf — Reverse proxy with Basic Auth for Ollama API # Generate .htpasswd: htpasswd -c nginx/.htpasswd devops # Set proxy_read_timeout high — streaming inference takes time server { listen 80; server_name _; # Basic auth — replace with mTLS for higher-security environments auth_basic "Ollama API"; auth_basic_user_file /etc/nginx/.htpasswd; location / { proxy_pass http://ollama:11434; proxy_http_version 1.1; # Critical: streaming responses require these headers proxy_set_header Connection ''; proxy_buffering off; proxy_cache off; # Increase timeouts — default 60s causes 504 on long generations proxy_read_timeout 300s; proxy_connect_timeout 10s; proxy_send_timeout 300s; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } # Health endpoint — exempt from auth for monitoring tools location /api/tags { auth_basic off; proxy_pass http://ollama:11434/api/tags; } } ``` For model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: `codellama:7b` in Q4_K_M for quick completions and inline code suggestions, and `mistral:7b-instruct-v0.2-q4_K_M` for longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they're genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM. We also pin models using digest references in our Modelfiles — `FROM llama3:8b-instruct@sha256:abc123...` — so a weekly `ollama pull` cron job doesn't silently swap out a model we've tuned prompts for. Model versioning matters more than most people realize when you're building automation on top of LLM output. For observability, Ollama has no native Prometheus endpoint, so we rely on `ollama ps` and `ollama list` in a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It's not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a loading state. More details on our homelab monitoring approach are over at [kuryzhev.cloud](https://kuryzhev.cloud/). Running a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first — hopefully this saves you the OOM crashes.