{"slug": "how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui", "title": "How to Build a Self-Hosted AI Gateway With LiteLLM and Open WebUI", "summary": "To build a self-hosted AI gateway using LiteLLM and Open WebUI to simplify management of multiple AI providers. By placing LiteLLM as a single gateway between user interfaces and various AI services (OpenAI, Anthropic, Groq, Ollama), the system allows providers to be swapped or added without rewriting application code. The open-source implementation can be deployed in under 30 minutes using Docker Compose, with configuration files for routing, authentication, and provider credentials.", "body_md": "If you've ever self-hosted AI tools, you know how quickly things get messy.\n\nOne app talks to OpenAI. Another uses Anthropic. You spin up Ollama locally and now there's a third endpoint to manage. Authentication is different everywhere. Switching models means rewriting integration code. And before long, you're spending more time maintaining glue code than actually building anything.\n\nI ran into this exact problem — so I built a cleaner setup.\n\n**The idea is simple:** put a single gateway in front of every provider, so the rest of your stack only ever talks to one API.\n\nI open-sourced the full working implementation here:\n\nClone it. Run `docker compose up`\n\n. You'll have a working AI gateway in under 30 minutes.\n\n## What This Stack Does\n\n-\n**One API**→ OpenAI, Anthropic, Groq, and Ollama all behind a single OpenAI-compatible endpoint -\n**One frontend**→ Open WebUI as the unified chat interface -\n**Secure remote access**→ Cloudflare Tunnel, no exposed ports, no open firewall rules -\n**Easy to maintain**→ providers can change underneath without touching your apps\n\nThe full setup takes roughly **20–45 minutes** depending on Docker image downloads and whether you already have local Ollama models installed.\n\n## The Stack\n\nNothing exotic here — just well-composed tools:\n\n| Component | Role |\n|---|---|\nLiteLLM |\nGateway / routing layer |\nOpen WebUI |\nChat frontend |\nPostgreSQL |\nState + metadata |\nDocker Compose |\nOrchestration |\nCloudflare Tunnel |\nSecure remote exposure |\n\n## Architecture\n\n```\nUser\n  ↓\nOpen WebUI\n  ↓\nLiteLLM (gateway)\n  ↓\nOpenAI / Anthropic / Groq / Ollama\n```\n\nThe key insight: your apps and frontend only talk to LiteLLM. Providers become interchangeable underneath. Add a new model, swap a provider, change routing — nothing else needs to know.\n\n## Who This Is For\n\n- Developers experimenting with local AI infrastructure\n- Teams consolidating multiple providers behind one API layer\n- Engineers building internal AI tooling\n- Anyone tired of maintaining separate provider integrations\n\nIf you've ever thought \"there has to be a simpler way to manage all these AI endpoints\" — this is that simpler way.\n\n## Why This Stack Exists\n\nMost self-hosted AI environments become hard to manage surprisingly fast. One application talks directly to OpenAI. Another uses Anthropic separately. Local Ollama models need their own endpoints. Authentication is inconsistent, and model switching slowly turns into infrastructure sprawl.\n\nBy placing LiteLLM in front of every provider, the rest of your system only needs to understand one interface. Providers can change, local models can be added, routing logic can evolve — without rewriting frontend or application logic every time.\n\n## Prerequisites\n\nBefore starting containers, make sure you have:\n\n- Docker Desktop\n- Docker Compose\n`curl`\n\n`cloudflared`\n\n- Ollama (optional, for local models)\n\nQuick verification:\n\n```\ndocker --version\ndocker compose version\ncloudflared --version\ncurl --version\n```\n\nIf using local Ollama models:\n\n```\nollama list\n```\n\nIf installed models appear, local inference is ready.\n\n## Repository Structure\n\nThe repo is intentionally lightweight:\n\n```\n├── Docker-compose.yml\n├── litellm-config.yml\n└── .env\n```\n\nEach file has a distinct job: Docker Compose orchestrates services, LiteLLM config handles routing and model aliases, `.env`\n\nstores secrets and runtime configuration.\n\n## Setting Up Environment Variables\n\nYour `.env`\n\nfile is where provider credentials live. Create or update it in the project root:\n\n```\nLITELLM_MASTER_KEY=sk-very-strong-key\nOPENAI_API_KEY=...\nANTHROPIC_API_KEY=...\nGROQ_API_KEY=...\nOLLAMA_CLOUD_API_BASE=https://<host>/v1\nOLLAMA_CLOUD_API_KEY=...\n```\n\nA few things that matter more than people expect:\n\n-\n`LITELLM_MASTER_KEY`\n\nbecomes the authentication layer between Open WebUI and LiteLLM - These values should\n**never** be committed into Git - Weak keys become a real security problem once remote access exists\n\nEven if this starts as a personal setup, treat the environment config like production from day one.\n\n## Wiring Docker Compose\n\nThe goal here is making sure the containers can talk to each other. Most deployment failures come from small config mismatches rather than Docker itself.\n\nYour `Docker-compose.yml`\n\nshould point Open WebUI directly at LiteLLM:\n\n```\nOPENAI_API_BASE_URL=http://litellm:4000/v1\nOPENAI_API_KEY=${LITELLM_MASTER_KEY}\n```\n\nThis tells Open WebUI where the gateway lives and which key to use.\n\nLiteLLM should mount the configuration file correctly:\n\n```\n./litellm-config.yml:/app/config.yaml\n```\n\n⚠️ That filename matters more than it looks. A typo here can cause Docker to create a directory instead of mounting the file — which leads to confusing startup errors later.\n\n## Configuring LiteLLM\n\nInside `litellm-config.yml`\n\n, LiteLLM defines model aliases, provider routing, gateway behavior, and authentication settings.\n\nThe most important section is the master key:\n\n```\ngeneral_settings:\n  master_key: os.environ/LITELLM_MASTER_KEY\n```\n\nThis keeps secrets outside the YAML file and makes credential rotation easier later.\n\nInside `model_list`\n\n, make sure provider model IDs are current. Model names change more frequently than most people expect — especially across Groq and newer OpenAI releases.\n\n## Starting the Stack\n\nOnce your config looks right, start everything:\n\n```\ndocker compose up -d --force-recreate\n```\n\nThe initial startup may take a minute while Docker pulls images, initializes PostgreSQL, and creates container state.\n\nVerify container health:\n\n```\ndocker ps --format 'table {{.Names}}\\t{{.Status}}\\t{{.Ports}}'\n```\n\nYou should see `open-webui`\n\n, `litellm`\n\n, and `litellm-db`\n\nall running.\n\nIf a container exits immediately, check its logs before moving forward:\n\n```\ndocker logs <container-name>\n```\n\n## Validating the Gateway\n\nBefore touching Open WebUI, validate LiteLLM first. The `/v1/models`\n\nendpoint confirms authentication works, providers loaded correctly, and model routing initialized.\n\n```\nset -a && source .env && \\\ncurl -s http://localhost:4000/v1/models \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\"\n```\n\nFor readable output:\n\n```\nset -a && source .env && \\\ncurl -s http://localhost:4000/v1/models \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\" \\\n  | python3 -m json.tool | head -n 80\n```\n\nIf this endpoint fails, Open WebUI will almost certainly fail too — so resolve gateway issues first.\n\n## Verifying Open WebUI\n\nOnce LiteLLM responds correctly, open the interface:\n\n```\nhttp://localhost:3000\n```\n\nYou should be able to create chats, select models, and send prompts normally.\n\nIf the model dropdown is empty, LiteLLM authentication is usually the cause — mismatched master keys, stale model IDs, or invalid provider credentials.\n\n## Keeping Provider Models Current\n\nThis catches a lot of people off guard: provider model identifiers change more often than you'd think. A deployment that worked perfectly a few months ago can break because a provider deprecated a model name.\n\n### Checking Local Ollama Models\n\n```\nollama list\n```\n\n### Checking Groq Models\n\n```\nset -a && source .env && \\\ncurl -s https://api.groq.com/openai/v1/models \\\n  -H \"Authorization: Bearer $GROQ_API_KEY\" \\\n  -H \"Content-Type: application/json\"\n```\n\nAfter updating model IDs in `litellm-config.yml`\n\n, recreate the stack:\n\n```\ndocker compose up -d --force-recreate\n```\n\n## Secure Remote Access with Cloudflare Tunnel\n\nAt this point, the stack only exists locally. The next step is exposing Open WebUI to the internet — without opening inbound ports, exposing your home IP, or managing reverse proxies manually.\n\nCloudflare Tunnel creates an outbound encrypted connection from your machine to Cloudflare's edge. You get:\n\n- Automatic HTTPS\n- Hidden origin infrastructure\n- Cloudflare proxy protection\n- Simple DNS management\n\n### Move DNS to Cloudflare\n\n- Add your domain to Cloudflare\n- Update nameservers at your registrar\n- Wait for propagation\n\n### Authenticate cloudflared\n\n```\ncloudflared tunnel login\n```\n\nThis opens a browser window for authorization.\n\n### Create the Tunnel\n\n```\ncloudflared tunnel create openwebui\n```\n\nCloudflare generates a tunnel UUID and a credentials JSON file.\n\n### Route a Subdomain\n\nAssuming your domain is `chat.yourdomain.tech`\n\n:\n\n```\ncloudflared tunnel route dns openwebui chat.yourdomain.tech\n```\n\n### Create the Tunnel Configuration\n\nCreate `~/.cloudflared/config.yml`\n\n:\n\n```\ntunnel: openwebui\ncredentials-file: ~/.cloudflared/<tunnel-uuid>.json\n\ningress:\n  - hostname: chat.yourdomain.tech\n    service: http://localhost:3000\n  - service: http_status:404\n```\n\nReplace `<tunnel-uuid>`\n\nwith the generated filename.\n\n### Run the Tunnel\n\n```\ncloudflared tunnel run openwebui\n```\n\nFor persistent startup on macOS:\n\n```\ncloudflared service install\ncloudflared service start\n```\n\nRunning the tunnel as a background service is significantly more reliable than keeping it in a terminal session.\n\n## Verifying Remote Access\n\nQuick DNS and connectivity check:\n\n```\ndig +short chat.yourdomain.tech\ncurl -I https://chat.yourdomain.tech\n```\n\nAlways use HTTPS for remote access. Cloudflare Tunnel is designed around secure proxied traffic.\n\n## Operational Health Checks\n\nOnce the stack is stable, this single command gives you a quick overview of everything:\n\n```\nset -a && source .env && \\\necho \"== Docker services ==\" && \\\ndocker ps --format 'table {{.Names}}\\t{{.Status}}\\t{{.Ports}}' && \\\necho \"\\n== Local Ollama models ==\" && \\\nollama list && \\\necho \"\\n== Groq model count ==\" && \\\ncurl -s https://api.groq.com/openai/v1/models \\\n  -H \"Authorization: Bearer $GROQ_API_KEY\" \\\n  -H \"Content-Type: application/json\" \\\n  | python3 -c 'import sys,json; d=json.load(sys.stdin); print(len(d.get(\"data\", [])))' && \\\necho \"\\n== LiteLLM models ==\" && \\\ncurl -s http://localhost:4000/v1/models \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\"\n```\n\nEven for personal deployments, having this kind of visibility saves a lot of debugging time.\n\n## Troubleshooting\n\nStable deployments drift. Provider APIs change, Docker mounts break, credentials expire, model IDs get deprecated. Here are the most common failure patterns.\n\n### Open WebUI Loads but Models Are Missing\n\nEmpty dropdowns, missing providers, or authentication errors in LiteLLM logs.\n\n```\ndocker logs --tail 200 litellm\n```\n\nVerify model visibility directly:\n\n```\nset -a && source .env && \\\ncurl -s http://localhost:4000/v1/models \\\n  -H \"Authorization: Bearer $LITELLM_MASTER_KEY\"\n```\n\nTypical fixes:\n\n- Verify\n`OPENAI_API_KEY=${LITELLM_MASTER_KEY}`\n\n- Confirm\n`master_key`\n\nuses the environment variable - Recreate containers:\n\n```\ndocker compose up -d --force-recreate\ndocker compose restart open-webui\n```\n\n### LiteLLM Fails with IsADirectoryError\n\nDocker accidentally created a directory instead of mounting the YAML file.\n\n```\nls -la ./litellm-config.yml ./litellm-config.yaml\ngrep -n \"litellm-config\" Docker-compose.yml\n```\n\nCorrect mount:\n\n```\n./litellm-config.yml:/app/config.yaml\n```\n\nThen recreate:\n\n```\ndocker compose up -d --force-recreate litellm\n```\n\n### Works Locally but Not Through Cloudflare\n\nIf local access works but the public hostname fails, focus on the tunnel:\n\n```\ncloudflared tunnel list\ncloudflared tunnel info openwebui\ncat ~/.cloudflared/config.yml\ndig +short chat.yourdomain.tech\ncurl -I https://chat.yourdomain.tech\n```\n\nMost remote-access failures come from inactive tunnel connectors, incorrect ingress targets, missing proxied DNS records, or running `cloudflared`\n\nin a temporary terminal session.\n\n### Models Appear but Generation Fails\n\nIf `/v1/models`\n\nworks but prompts fail — provider credentials may be invalid, quotas exhausted, or model IDs no longer exist.\n\n```\nset -a && source .env && \\\nenv | grep -E '^(OPENAI_API_KEY|GROQ_API_KEY|ANTHROPIC_API_KEY|LITELLM_MASTER_KEY)=' \\\n  | sed 's/=.*/=<set>/'\n```\n\nThen inspect LiteLLM logs:\n\n```\ndocker logs --tail 300 litellm\n```\n\nRefreshing provider model IDs solves this surprisingly often.\n\n## Security Recommendations\n\nOnce remote access exists, basic hardening matters:\n\n- Use a strong\n`LITELLM_MASTER_KEY`\n\n- Don't expose LiteLLM directly to the internet\n- Rotate provider API keys periodically\n- Keep CORS rules restrictive\n\nFor private or team usage, **Cloudflare Access** adds identity-aware access control in front of Open WebUI — worth enabling.\n\n## Capture a Known-Good Baseline\n\nOnce things are stable, save:\n\n-\n`docker ps`\n\noutput -\n`/v1/models`\n\noutput - Active model aliases\n- Tunnel status:\n\n```\ncloudflared tunnel info openwebui\n```\n\nWhen something breaks later, comparing against a working snapshot is almost always faster than debugging from scratch.\n\n## Try It Yourself\n\nThe full working implementation — Docker Compose, LiteLLM config, environment setup — is all here:\n\nClone it, add your API keys, run `docker compose up`\n\n, and you'll have a working AI gateway.\n\nIf you found this useful, ⭐ the repo — it helps more people find it.\n\nGot questions or improvements? Open an issue or drop a comment below. I'm actively maintaining this.\n\n*Originally published on HackerNoon.*", "url": "https://wpnews.pro/news/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui", "canonical_source": "https://dev.to/khalifornia/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui-fn3", "published_at": "2026-05-21 15:19:09+00:00", "updated_at": "2026-05-21 15:34:34.102772+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "open-source", "developer-tools"], "entities": ["LiteLLM", "Open WebUI", "OpenAI", "Anthropic", "Groq", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui", "markdown": "https://wpnews.pro/news/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui.md", "text": "https://wpnews.pro/news/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui.txt", "jsonld": "https://wpnews.pro/news/how-to-build-a-self-hosted-ai-gateway-with-litellm-and-open-webui.jsonld"}}