{"slug": "self-hosted-ollama-homelab-3-mistakes-running-local-llms", "title": "Self-Hosted Ollama Homelab: 3 Mistakes Running Local LLMs", "summary": "A developer's team encountered three critical mistakes while setting up a self-hosted Ollama homelab for DevOps assistance: silent CPU fallback due to CUDA version mismatch, an OOM crash that took down a Gitea container, and an exposed API endpoint. The team resolved the GPU acceleration issue by updating CUDA to 11.8 and pinning nvidia-container-toolkit, and recommends verifying GPU offload with OLLAMA_DEBUG=1.", "body_md": "*Originally published on kuryzhev.cloud*\n\nWe thought setting up a self-hosted Ollama homelab for DevOps assistance would take an afternoon. Three OOM crashes, one exposed API endpoint, and a silent CPU fallback later, here's what actually works — and what we'd do differently from day one.\n\nOur homelab runs a fairly typical self-hosted DevOps stack: Gitea for source control, Drone CI for pipelines, Proxmox for VM management, and a handful of Docker Compose services scattered across two physical hosts. Day-to-day tasks involve writing Ansible playbooks, debugging systemd units, generating Dockerfiles, and occasionally reverse-engineering someone else's Terraform. All of that involves a lot of back-and-forth with language models.\n\nThe problem with cloud-based LLM APIs — ChatGPT, Claude, Gemini — is that they require pasting real infrastructure configs into a browser. Hostnames, internal IP ranges, secrets that slipped into a playbook comment, service account names. None of that should leave the network. We also have air-gapped lab segments that physically can't reach the internet, and we wanted consistent tooling across both environments.\n\nWe chose [Ollama](https://ollama.com/) for a few concrete reasons: it ships as a single binary, it manages model downloads via a clean CLI, and it exposes an OpenAI-compatible REST API on port 11434 out of the box. That last point matters — it means any tool that supports the OpenAI API (Continue.dev in VS Code, shell scripts using `curl`\n\n, custom Python utilities) works against Ollama without modification. The plan was straightforward. The execution was not.\n\nThe first host we tried had an NVIDIA GTX 1060 3GB installed. We ran `ollama run llama3`\n\n, watched it load, and immediately noticed something was wrong. Inference was crawling at roughly 2 tokens per second. We expected 25 or more. No error. No warning. The model just loaded silently into system RAM and ran entirely on CPU.\n\nWe burned two hours checking the wrong things — model quantization, RAM speed, thermal throttling — before running `ollama ps`\n\nand seeing `100% CPU`\n\nnext to the loaded model. Then we checked the logs with `OLLAMA_DEBUG=1 ollama serve`\n\nand found the real message:\n\n```\nmsg=\"no GPU detected\"\n```\n\nThe GTX 1060 was visible to `nvidia-smi`\n\n. The driver was loaded. But Ollama requires CUDA 11.8 or higher, and that host was running CUDA 11.4. The mismatch was silent from the client's perspective — Ollama simply fell back to CPU without surfacing any error to the caller.\n\n**Watch out for this:** `nvidia-smi`\n\nshowing your GPU does not mean Ollama will use it. Always verify with `OLLAMA_DEBUG=1`\n\nand look for `offload layers: 32/32`\n\nin the output. That line confirms full GPU acceleration. Anything less means partial or zero offload.\n\nThe fix required three steps: updating the CUDA toolkit to 11.8, pinning `nvidia-container-toolkit`\n\nto version 1.14.3 (earlier versions had inconsistent behavior with Docker runtime detection), and ensuring `/etc/docker/daemon.json`\n\nhad `\"default-runtime\": \"nvidia\"`\n\nset. After that, the same model ran at 28 tokens per second on the same hardware. Same model, same host, completely different experience.\n\nWe also verified the CUDA version properly this time:\n\n```\n# Check CUDA version — must be >= 11.8 for Ollama GPU support\nnvcc --version\n\n# Confirm Docker is using the nvidia runtime\ndocker info | grep -i runtime\n\n# Verify GPU offload after starting Ollama with debug logging\nOLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i \"offload layers\"\n```\n\nOnce GPU acceleration was working, we got greedy and pulled a larger model. `codellama:13b`\n\nseemed like the right tool for generating longer Ansible roles and reviewing Dockerfiles with more context. We pulled it, ran it, and about forty seconds into the first inference request, everything went sideways.\n\nThe OOM killer fired. It took down our Gitea container mid-push. A developer on the team lost a commit they hadn't pushed yet. The `codellama:13b`\n\nmodel in Q4_K_M quantization needs roughly 8.5GB of VRAM — and on a 16GB host already running a dozen services, the model load spiked total RAM consumption to 14GB instantaneously. The kernel picked Gitea as the most expendable process. It wasn't.\n\nThe root cause was straightforward: we were running Ollama directly on the host with no cgroup constraints, no memory limits, and no model eviction policy. In Ollama v0.1.38 and later, `OLLAMA_MAX_LOADED_MODELS`\n\ncontrols how many models stay resident in memory simultaneously. In earlier versions, there's no eviction control at all. We hadn't set either `OLLAMA_MAX_LOADED_MODELS`\n\nor `OLLAMA_NUM_PARALLEL`\n\n, so Ollama defaulted to greedy behavior — loading whatever was asked, keeping it resident, and accepting parallel requests that multiplied memory pressure.\n\n**Watch out for this:** `OLLAMA_NUM_PARALLEL`\n\ndefaults to 4 in recent versions. On a constrained homelab host, four simultaneous inference requests against a 7B model will absolutely OOM your other services. Set it to 1 unless you have dedicated hardware.\n\nThe correct approach was to containerize Ollama with explicit resource caps and isolate it from critical services on a dedicated Docker network. We moved everything into Docker Compose, set `mem_limit: 10g`\n\n, and added environment variables to enforce single-model, single-request behavior.\n\nThis one is embarrassing to admit. The default `ollama serve`\n\nbinds to `127.0.0.1`\n\n, which is fine for bare-metal installs. But our Docker Compose setup used `ports: \"11434:11434\"`\n\nwithout thinking through what that means on a flat home network. It published the port to every interface on the host — which sits on the same physical switch as our IoT VLAN due to a pfSense VLAN rule we'd misconfigured months earlier.\n\nOllama has no built-in authentication as of v0.1.x. Anyone who could reach port 11434 could call `GET /api/tags`\n\nto enumerate loaded models, send inference requests, or — and this is the one that actually worried us — trigger a `POST /api/pull`\n\nto download a multi-gigabyte model and exhaust disk space. There's no rate limiting either. A single unauthenticated pull request from a compromised IoT device could fill a volume.\n\nWe also had `OLLAMA_HOST=0.0.0.0`\n\nset inside the container, which is necessary for the container's internal binding but easy to misread as \"only affects the container namespace.\" It doesn't change the fact that the Docker port mapping exposes it externally.\n\nThe fix was an Nginx reverse proxy with HTTP Basic Auth sitting in front of Ollama, combined with binding the proxy's published port to `127.0.0.1:8080`\n\non the host. Remote access goes through Tailscale, which restricts it to trusted nodes only. For the health check endpoint used by monitoring tools, we carved out an auth-exempt `location /api/tags`\n\nblock in Nginx.\n\nThe current setup avoids all three mistakes. Here's the full Docker Compose configuration we run in production on the homelab. It uses a pinned Ollama image, GPU passthrough via the `nvidia`\n\nruntime, hard memory caps, and an Nginx auth proxy — all wired together with a named volume for model persistence.\n\nThis Docker Compose file defines the full hardened stack. Read the inline comments carefully — several of the environment variables are non-obvious in their effect:\n\n```\n# docker-compose.yml — Production-hardened Ollama stack for homelab\n# Tested with Ollama v0.1.38, Docker 24.x, nvidia-container-toolkit 1.14.3\n\nversion: \"3.9\"\n\nservices:\n  ollama:\n    image: ollama/ollama:0.1.38          # pinned — avoid surprise API changes\n    container_name: ollama\n    restart: unless-stopped\n    runtime: nvidia                       # requires nvidia-container-toolkit\n    environment:\n      - OLLAMA_HOST=0.0.0.0              # bind inside container; proxy handles external auth\n      - OLLAMA_NUM_PARALLEL=1            # prevent concurrent request memory spikes\n      - OLLAMA_MAX_LOADED_MODELS=1       # evict previous model before loading new one\n      - OLLAMA_DEBUG=0                   # set to 1 temporarily to verify GPU offload layers\n    volumes:\n      - ollama_models:/root/.ollama      # persist models across container rebuilds\n    networks:\n      - ollama_internal                  # isolated from critical services network\n    deploy:\n      resources:\n        limits:\n          memory: 10g                    # hard cap — prevents OOM-killing neighbors\n        reservations:\n          devices:\n            - driver: nvidia\n              count: 1\n              capabilities: [gpu]\n    healthcheck:\n      test: [\"CMD\", \"curl\", \"-f\", \"http://localhost:11434/api/tags\"]\n      interval: 30s\n      timeout: 10s\n      retries: 3\n      start_period: 20s                  # model server takes ~15s to initialize\n\n  ollama-proxy:\n    image: nginx:1.25-alpine\n    container_name: ollama-proxy\n    restart: unless-stopped\n    ports:\n      - \"127.0.0.1:8080:80\"             # bind to loopback only; use Tailscale for remote access\n    volumes:\n      - ./nginx/ollama.conf:/etc/nginx/conf.d/default.conf:ro\n      - ./nginx/.htpasswd:/etc/nginx/.htpasswd:ro\n    networks:\n      - ollama_internal\n    depends_on:\n      ollama:\n        condition: service_healthy\n\nvolumes:\n  ollama_models:\n    driver: local\n\nnetworks:\n  ollama_internal:\n    driver: bridge\n    internal: false                      # set true if Ollama should have no outbound internet\n```\n\nThe Nginx config below handles streaming inference correctly. The two most common mistakes people make here are forgetting `proxy_buffering off`\n\n(which causes the response to appear all at once instead of streaming) and leaving `proxy_read_timeout`\n\nat the default 60 seconds, which causes 504 errors mid-generation on longer prompts:\n\n```\n# nginx/ollama.conf — Reverse proxy with Basic Auth for Ollama API\n# Generate .htpasswd: htpasswd -c nginx/.htpasswd devops\n# Set proxy_read_timeout high — streaming inference takes time\n\nserver {\n    listen 80;\n    server_name _;\n\n    # Basic auth — replace with mTLS for higher-security environments\n    auth_basic           \"Ollama API\";\n    auth_basic_user_file /etc/nginx/.htpasswd;\n\n    location / {\n        proxy_pass         http://ollama:11434;\n        proxy_http_version 1.1;\n\n        # Critical: streaming responses require these headers\n        proxy_set_header   Connection '';\n        proxy_buffering    off;\n        proxy_cache        off;\n\n        # Increase timeouts — default 60s causes 504 on long generations\n        proxy_read_timeout    300s;\n        proxy_connect_timeout 10s;\n        proxy_send_timeout    300s;\n\n        proxy_set_header Host $host;\n        proxy_set_header X-Real-IP $remote_addr;\n    }\n\n    # Health endpoint — exempt from auth for monitoring tools\n    location /api/tags {\n        auth_basic off;\n        proxy_pass http://ollama:11434/api/tags;\n    }\n}\n```\n\nFor model selection, we settled on two workhorses that fit within 6GB VRAM with quantization: `codellama:7b`\n\nin Q4_K_M for quick completions and inline code suggestions, and `mistral:7b-instruct-v0.2-q4_K_M`\n\nfor longer reasoning tasks like reviewing a Terraform plan or explaining a failing systemd unit. The 13B models sound appealing but they're genuinely impractical on homelab hardware unless you have a dedicated GPU with 10GB+ VRAM.\n\nWe also pin models using digest references in our Modelfiles — `FROM llama3:8b-instruct@sha256:abc123...`\n\n— so a weekly `ollama pull`\n\ncron job doesn't silently swap out a model we've tuned prompts for. Model versioning matters more than most people realize when you're building automation on top of LLM output.\n\nFor observability, Ollama has no native Prometheus endpoint, so we rely on `ollama ps`\n\nand `ollama list`\n\nin a lightweight cron-based check, plus the Docker healthcheck defined in the Compose file. It's not elegant, but it catches the cases that actually happen: the model server failing to start, or a model getting stuck in a loading state. More details on our homelab monitoring approach are over at [kuryzhev.cloud](https://kuryzhev.cloud/).\n\nRunning a self-hosted Ollama homelab is genuinely useful once the stack is stable. The self-hosted Ollama homelab we have now handles dozens of DevOps assistance requests per day with zero data leaving the network, predictable resource usage, and a setup that survives container restarts and host reboots without manual intervention. Getting there required making all three of these mistakes first — hopefully this saves you the OOM crashes.", "url": "https://wpnews.pro/news/self-hosted-ollama-homelab-3-mistakes-running-local-llms", "canonical_source": "https://dev.to/oleksandr_kuryzhev_42873f/self-hosted-ollama-homelab-3-mistakes-running-local-llms-11l1", "published_at": "2026-06-26 07:02:10+00:00", "updated_at": "2026-06-26 07:34:04.824884+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["Ollama", "NVIDIA", "CUDA", "Gitea", "Drone CI", "Proxmox", "Docker", "Continue.dev"], "alternates": {"html": "https://wpnews.pro/news/self-hosted-ollama-homelab-3-mistakes-running-local-llms", "markdown": "https://wpnews.pro/news/self-hosted-ollama-homelab-3-mistakes-running-local-llms.md", "text": "https://wpnews.pro/news/self-hosted-ollama-homelab-3-mistakes-running-local-llms.txt", "jsonld": "https://wpnews.pro/news/self-hosted-ollama-homelab-3-mistakes-running-local-llms.jsonld"}}