I use Claude Code every day but not everything belongs in the public domain: personal data, private notes, half-formed ideas. For those I’d rather keep the conversation local.
So I also run a local assistant. A small Docker Compose stack gives me a private LLM with a chat UI that looks a lot like the commercial ones and for that kind of prompt it means no per-token bill and nothing leaving my local network.
The whole thing is two services: Ollama to run the models and Open WebUI to talk to them.
What you get #
Ollama pulls models, keeps them on disk, and serves them over a simple HTTP API on port11434
.Open WebUI is the front end. It’s a polished, self-hosted chat interface with conversations, multiple models, user accounts and file uploads. It talks to Ollama over the API, so you get a ChatGPT-style experience locally.
Two containers, one network, two volumes for persistence. That’s the entire design.
Before you start #
You need Docker which is likely already installed if you run anything in a homelab.
For anything beyond small models you also want an NVIDIA GPU and the NVIDIA Container Toolkit installed on the host. The toolkit is what lets a container see the GPU. Without it, the GPU block in the Compose file will fail to start, and you’ll want the CPU-only variant instead (I cover that later). You can check the toolkit is working with:
1
docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
If that prints your GPU, you’re ready. If it errors, fix the toolkit first, because the LLM stack relies on exactly that plumbing.
The Compose file #
Here’s the stack in full. I’ll break down the parts that matter underneath.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
services:
ollama:
image: ollama/ollama
command: serve
expose:
- 11434/tcp
healthcheck:
test: ollama --version || exit 1
volumes:
- ollama:/root/.ollama
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["all"]
capabilities: [gpu]
webui:
image: ghcr.io/open-webui/open-webui:main
ports:
- 8080:8080/tcp
environment:
- OLLAMA_BASE_URL=http://ollama:11434
volumes:
- open-webui:/app/backend/data
depends_on:
- ollama
volumes:
ollama:
open-webui:
Save it as docker-compose.yml
and run docker compose up -d
to bring the full stack online.
Before you do it’s worth understanding how it works in a little more detail below.
How the two services find each other
Notice OLLAMA_BASE_URL=http://ollama:11434
. That ollama
is not a hostname I’ve configured, it’s the service name for ollama. Compose gives every service a DNS entry on the default network for the project. So Open WebUI reaches the Ollama by name, over the internal network, with no host ports or IP addresses involved.
That’s also why Ollama uses expose
rather than ports
. expose
makes 11434
reachable to other containers on the same network but does not publish it to the host. The only thing I actually publish is Open WebUI’s 8080
, because that’s the one interface a human needs to reach. Keeping the raw model API off the host is a small but real bit of hygiene and I’ll come back to why it matters.
GPU passthrough, the part everyone gets stuck on
This block is what hands the GPU to the container:
1
2
3
4
5
6
7
deploy:
resources:
reservations:
devices:
- driver: nvidia
device_ids: ["all"]
capabilities: [gpu]
A few things worth knowing:
capabilities: [gpu]
is required. If you omit it, Docker will not grant GPU access even with the rest of the block present.device_ids: ["all"]
gives the container every GPU. If you have more than one and want to pin Ollama to a specific card, swap it for something likedevice_ids: ["0"]
.- This is the Compose equivalent of
docker run --gpus all
. It only works if the NVIDIA Container Toolkit from the previous section is installed. This is where “it won’t start” almost always traces back to.
Get this right and Ollama will load models into VRAM and run them on the GPU, which is the difference between a usable assistant and one you wait on.
Persistence and health
The two named volumes persist the models to stop re-down several gigabytes of model every time a container restarts:
ollama:/root/.ollama
keeps your pulled models.open-webui:/app/backend/data
keeps your accounts, chats and settings.
The healthcheck
on Ollama gives Docker a real readiness signal instead of “the process is running, probably”. It’s simple, but it means depends_on
and your own tooling can reason about whether the engine is actually up.
Bringing it up #
1
2
docker compose up -d
docker compose ps
Open WebUI is now on http://localhost:8080
. The first account you create becomes the admin, so do that before you expose it anywhere.
The stack is running, but Ollama has no models yet. Pull one:
1
docker compose exec ollama ollama pull llama3.1:8b
llama3.1:8b
is a good general starting point that fits comfortably on an 8 GB card. If you’re tight on VRAM, try a smaller model such as qwen2.5:3b
; if you’ve got headroom, pull something larger. Once it’s down, it shows up in the model picker in Open WebUI and you can start chatting. You can also pull models straight from the UI, but doing it once on the command line makes it obvious where the work happens.
No GPU? The CPU-only variant #
You don’t need a GPU to try this. Drop the entire deploy:
block from the ollama
service and everything else stays the same:
1
2
3
4
5
6
7
8
9
ollama:
image: ollama/ollama
command: serve
expose:
- 11434/tcp
healthcheck:
test: ollama --version || exit 1
volumes:
- ollama:/root/.ollama
It will run happily on CPU. Be realistic about it, a small model will feel sluggish and a large one will feel broken. CPU-only is great for kicking the tyres and fine for light use with a 3B-class model. For anything you’ll use daily, the GPU is what makes it pleasant.
Don’t put this on the internet naked #
Open WebUI has its own authentication, which is good. It is still not something I’d publish straight to the internet on a raw port, and the Ollama API has no auth at all, which is exactly why the Compose file above never publishes 11434
to the host.
If you want to reach this from outside the house, put it behind a reverse proxy that terminates TLS and ideally, an auth layer in front (I run mine local only with no access from outside my home network).
Is running it yourself actually worth it? #
I find that it is worth it for me with one real caveat. Here’s the full trade-off:
Cost. If you already own the required hardware then its essentially free. No per-token billing, no surprise invoice.Privacy. Nothing leaves your network. Your prompts are yours.Latency. On a decent GPU, first-token latency is genuinely competitive, and there’s no network round trip to a provider.Model quality. The open models you can run at home are very good and getting better, but the largest frontier models still have an edge on the more complex tasks. For summarising, drafting, coding help and general questions, local models are more than good enough.
For me personally, the cloud tools stay in the mix for the heavy, cutting-edge work, and anything personal or private goes to the local model instead, where the traffic never leaves the house. The simple stack above is what makes it a five-minute decision rather than a project.
Wrapping up #
Two services, one Compose file, and you’ve got a private ChatGPT-like prompt running on your own hardware. It has very simple containers to setup: point Open WebUI at Ollama by its service name, hand the GPU over with the deploy
block (and the NVIDIA toolkit behind it), keep your models and chats in named volumes, and never publish the inference API. Get those right and the rest is just picking which models to pull.