{"slug": "run-your-own-self-hosted-llms-with-docker-compose", "title": "Run your own self-hosted LLMs with Docker Compose", "summary": "A developer published a guide for running self-hosted large language models locally using Docker Compose with Ollama and Open WebUI, enabling a private ChatGPT-like experience without sending data to external servers. The setup requires Docker and optionally an NVIDIA GPU with the Container Toolkit for hardware acceleration.", "body_md": "# Run your own ChatGPT: self-hosted LLMs with Docker Compose\n\nI use Claude Code every day but not everything belongs in the public domain: personal data, private notes, half-formed ideas. For those I’d rather keep the conversation local.\n\nSo I also run a local assistant. A small Docker Compose stack gives me a private LLM with a chat UI that looks a lot like the commercial ones and for that kind of prompt it means no per-token bill and nothing leaving my local network.\n\nThe whole thing is two services: [Ollama](https://ollama.com/) to run the models and [Open WebUI](https://openwebui.com/) to talk to them.\n\n## What you get\n\n**Ollama** pulls models, keeps them on disk, and serves them over a simple HTTP API on port`11434`\n\n.**Open WebUI** is the front end. It’s a polished, self-hosted chat interface with conversations, multiple models, user accounts and file uploads. It talks to Ollama over the API, so you get a ChatGPT-style experience locally.\n\nTwo containers, one network, two volumes for persistence. That’s the entire design.\n\n## Before you start\n\nYou need Docker which is likely already installed if you run anything in a homelab.\n\nFor anything beyond small models you also want an NVIDIA GPU and the **NVIDIA Container Toolkit** installed on the host. The toolkit is what lets a container see the GPU. Without it, the GPU block in the Compose file will fail to start, and you’ll want the CPU-only variant instead (I cover that later). You can check the toolkit is working with:\n\n```\n1\ndocker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi\n```\n\nIf that prints your GPU, you’re ready. If it errors, fix the toolkit first, because the LLM stack relies on exactly that plumbing.\n\n## The Compose file\n\nHere’s the stack in full. I’ll break down the parts that matter underneath.\n\n```\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\n31\n32\nservices:\n  ollama:\n    image: ollama/ollama\n    command: serve\n    expose:\n      - 11434/tcp\n    healthcheck:\n      test: ollama --version || exit 1\n    volumes:\n      - ollama:/root/.ollama\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              device_ids: [\"all\"]\n              capabilities: [gpu]\n\n  webui:\n    image: ghcr.io/open-webui/open-webui:main\n    ports:\n      - 8080:8080/tcp\n    environment:\n      - OLLAMA_BASE_URL=http://ollama:11434\n    volumes:\n      - open-webui:/app/backend/data\n    depends_on:\n      - ollama\n\nvolumes:\n  ollama:\n  open-webui:\n```\n\nSave it as `docker-compose.yml`\n\nand run `docker compose up -d`\n\nto bring the full stack online.\n\nBefore you do it’s worth understanding how it works in a little more detail below.\n\n### How the two services find each other\n\nNotice `OLLAMA_BASE_URL=http://ollama:11434`\n\n. That `ollama`\n\nis not a hostname I’ve configured, it’s the service name for ollama. Compose gives every service a DNS entry on the default network for the project. So Open WebUI reaches the Ollama by name, over the internal network, with no host ports or IP addresses involved.\n\nThat’s also why Ollama uses `expose`\n\nrather than `ports`\n\n. `expose`\n\nmakes `11434`\n\nreachable to other containers on the same network but does **not** publish it to the host. The only thing I actually publish is Open WebUI’s `8080`\n\n, because that’s the one interface a human needs to reach. Keeping the raw model API off the host is a small but real bit of hygiene and I’ll come back to why it matters.\n\n### GPU passthrough, the part everyone gets stuck on\n\nThis block is what hands the GPU to the container:\n\n```\n1\n2\n3\n4\n5\n6\n7\n    deploy:\n      resources:\n        reservations:\n          devices:\n            - driver: nvidia\n              device_ids: [\"all\"]\n              capabilities: [gpu]\n```\n\nA few things worth knowing:\n\n`capabilities: [gpu]`\n\nis required. If you omit it, Docker will not grant GPU access even with the rest of the block present.`device_ids: [\"all\"]`\n\ngives the container every GPU. If you have more than one and want to pin Ollama to a specific card, swap it for something like`device_ids: [\"0\"]`\n\n.- This is the Compose equivalent of\n`docker run --gpus all`\n\n. It only works if the NVIDIA Container Toolkit from the previous section is installed. This is where “it won’t start” almost always traces back to.\n\nGet this right and Ollama will load models into VRAM and run them on the GPU, which is the difference between a usable assistant and one you wait on.\n\n### Persistence and health\n\nThe two named volumes persist the models to stop re-downloading several gigabytes of model every time a container restarts:\n\n`ollama:/root/.ollama`\n\nkeeps your pulled models.`open-webui:/app/backend/data`\n\nkeeps your accounts, chats and settings.\n\nThe `healthcheck`\n\non Ollama gives Docker a real readiness signal instead of “the process is running, probably”. It’s simple, but it means `depends_on`\n\nand your own tooling can reason about whether the engine is actually up.\n\n## Bringing it up\n\n```\n1\n2\ndocker compose up -d\ndocker compose ps\n```\n\nOpen WebUI is now on `http://localhost:8080`\n\n. The first account you create becomes the admin, so do that before you expose it anywhere.\n\nThe stack is running, but Ollama has no models yet. Pull one:\n\n```\n1\ndocker compose exec ollama ollama pull llama3.1:8b\n```\n\n`llama3.1:8b`\n\nis a good general starting point that fits comfortably on an 8 GB card. If you’re tight on VRAM, try a smaller model such as `qwen2.5:3b`\n\n; if you’ve got headroom, pull something larger. Once it’s down, it shows up in the model picker in Open WebUI and you can start chatting. You can also pull models straight from the UI, but doing it once on the command line makes it obvious where the work happens.\n\n## No GPU? The CPU-only variant\n\nYou don’t need a GPU to try this. Drop the entire `deploy:`\n\nblock from the `ollama`\n\nservice and everything else stays the same:\n\n```\n1\n2\n3\n4\n5\n6\n7\n8\n9\n  ollama:\n    image: ollama/ollama\n    command: serve\n    expose:\n      - 11434/tcp\n    healthcheck:\n      test: ollama --version || exit 1\n    volumes:\n      - ollama:/root/.ollama\n```\n\nIt will run happily on CPU. Be realistic about it, a small model will feel sluggish and a large one will feel broken. CPU-only is great for kicking the tyres and fine for light use with a 3B-class model. For anything you’ll use daily, the GPU is what makes it pleasant.\n\n## Don’t put this on the internet naked\n\nOpen WebUI has its own authentication, which is good. It is still not something I’d publish straight to the internet on a raw port, and the Ollama API has no auth at all, which is exactly why the Compose file above never publishes `11434`\n\nto the host.\n\nIf you want to reach this from outside the house, put it behind a reverse proxy that terminates TLS and ideally, an auth layer in front (I run mine local only with no access from outside my home network).\n\n## Is running it yourself actually worth it?\n\nI find that it is worth it for me with one real caveat. Here’s the full trade-off:\n\n**Cost.** If you already own the required hardware then its essentially free. No per-token billing, no surprise invoice.**Privacy.** Nothing leaves your network. Your prompts are yours.**Latency.** On a decent GPU, first-token latency is genuinely competitive, and there’s no network round trip to a provider.**Model quality.** The open models you can run at home are very good and getting better, but the largest frontier models still have an edge on the more complex tasks. For summarising, drafting, coding help and general questions, local models are more than good enough.\n\nFor me personally, the cloud tools stay in the mix for the heavy, cutting-edge work, and anything personal or private goes to the local model instead, where the traffic never leaves the house. The simple stack above is what makes it a five-minute decision rather than a project.\n\n## Wrapping up\n\nTwo services, one Compose file, and you’ve got a private ChatGPT-like prompt running on your own hardware. It has very simple containers to setup: point Open WebUI at Ollama by its service name, hand the GPU over with the `deploy`\n\nblock (and the NVIDIA toolkit behind it), keep your models and chats in named volumes, and never publish the inference API. Get those right and the rest is just picking which models to pull.", "url": "https://wpnews.pro/news/run-your-own-self-hosted-llms-with-docker-compose", "canonical_source": "https://totaldebug.uk/posts/self-hosted-llms-ollama-open-webui-docker-compose/", "published_at": "2026-07-08 07:22:46+00:00", "updated_at": "2026-07-08 07:29:44.851220+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "ai-infrastructure", "developer-tools"], "entities": ["Ollama", "Open WebUI", "Docker", "NVIDIA", "Claude Code"], "alternates": {"html": "https://wpnews.pro/news/run-your-own-self-hosted-llms-with-docker-compose", "markdown": "https://wpnews.pro/news/run-your-own-self-hosted-llms-with-docker-compose.md", "text": "https://wpnews.pro/news/run-your-own-self-hosted-llms-with-docker-compose.txt", "jsonld": "https://wpnews.pro/news/run-your-own-self-hosted-llms-with-docker-compose.jsonld"}}