# Deploying LiteLLM: An Open-Source AI Gateway

> Source: <https://dev.to/vultr/deploying-litellm-an-open-source-ai-gateway-2idp>
> Published: 2026-09-23 19:19:19+00:00

[LiteLLM](https://litellm.ai) is an open-source AI gateway that provides a unified, OpenAI-compatible API for over 100 large language model (LLM) providers, removing the need to integrate with each provider's SDK and authentication scheme. In production, teams use LiteLLM to issue virtual keys with budget limits, track token usage and spend across providers, and configure centralized access control and routing. This guide walks through deploying LiteLLM on a Linux server using Docker Compose with PostgreSQL for persistent storage, Prometheus for metrics collection, and Traefik as the reverse proxy for HTTPS access. By the end, you'll have a fully functional, OpenAI-compatible AI gateway accessible over a custom domain, with virtual key management, spend tracking, and provider routing configured.

Before you begin, you need a Linux-based server with at least 4 CPU cores and 8 GB of RAM as a non-root user with sudo privileges, Docker and Docker Compose installed, a DNS A record (such as `litellm.example.com`) pointing to your server's IP address, and an API key from at least one [supported LLM provider](https://docs.litellm.ai/docs/providers).

LiteLLM requires a configuration file to define model providers and routing rules, a Prometheus configuration file for metrics scraping, and environment variables for secrets and database credentials.

**1. Create the project directory with subdirectories for persistent data:**

``` bash
$ mkdir -p ~/litellm/{letsencrypt,postgres,prometheus}
```

`letsencrypt` stores SSL/TLS certificates, `postgres` persists PostgreSQL database files, and `prometheus` persists Prometheus metrics data.

**2. Navigate to the project directory:**

``` bash
$ cd ~/litellm
```

**3. Set matching ownership on the Prometheus host directory (Prometheus runs as UID 65534 inside the container):**

``` bash
$ sudo chown -R 65534:65534 prometheus
```

**4. Generate a master key and salt key for LiteLLM. Run this command twice to produce two separate values:**

``` bash
$ openssl rand -hex 32
```

Save both values for the `LITELLM_MASTER_KEY` and `LITELLM_SALT_KEY` fields in the next step.

**5. Create the `.env` file to store credentials and secrets:**

``` bash
$ nano .env
```

**6. Add the following values:**

```
DOMAIN=litellm.example.com
LETSENCRYPT_EMAIL=admin@example.com
LITELLM_MASTER_KEY=sk-YOUR_MASTER_KEY
LITELLM_SALT_KEY=sk-YOUR_SALT_KEY
LLM_PROVIDER_API_KEY=YOUR_LLM_PROVIDER_API_KEY
POSTGRES_PASSWORD=STRONG_DATABASE_PASSWORD
DATABASE_URL=postgresql://llmproxy:${POSTGRES_PASSWORD}@db:5432/litellm
UI_USERNAME=admin
UI_PASSWORD=YOUR_UI_PASSWORD
```

Replace the placeholders with your own values: `litellm.example.com` is the domain pointing to your server's IP address; `admin@example.com` is the email address for Let's Encrypt notifications; `sk-YOUR_MASTER_KEY` is the admin key used to authenticate with the LiteLLM API (replace `YOUR_MASTER_KEY` with the first generated value and keep the `sk-` prefix); `sk-YOUR_SALT_KEY` encrypts provider credentials stored in PostgreSQL and cannot be changed after the first model is added; `YOUR_LLM_PROVIDER_API_KEY` is the API key for the provider configured in `config.yaml`; `STRONG_DATABASE_PASSWORD` is the PostgreSQL password used by both the database container and the connection string; `admin`/` YOUR_UI_PASSWORD` are the LiteLLM dashboard credentials.

**7. Create the LiteLLM configuration file:**

``` bash
$ nano config.yaml
```

**8. Add the following contents:**

```
model_list:
  - model_name: my-model
    litellm_params:
      model: provider/model
      api_key: os.environ/LLM_PROVIDER_API_KEY

litellm_settings:
  callbacks:
    - prometheus
  require_auth_for_metrics_endpoint: true

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
```

Replace `my-model` with a name of your choice to identify this model within LiteLLM, and `provider/model` with the LiteLLM provider prefix followed by the model identifier (for example, `anthropic/claude-haiku-4-5`). The `os.environ/` prefix tells LiteLLM to read the value from an environment variable at runtime rather than hardcoding it. The `litellm_settings` block enables Prometheus metrics via the `callbacks` field, and `require_auth_for_metrics_endpoint: true` restricts the `/metrics` endpoint to authenticated requests, so only Prometheus (sending the master key as a Bearer token) can scrape metrics successfully. The `general_settings` block defines the master key LiteLLM uses to authenticate admin API requests and virtual key management operations.

**9. Create the Prometheus configuration file:**

``` bash
$ nano prometheus.yml
```

**10. Add the following contents:**

```
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "litellm"
    static_configs:
      - targets: ["litellm:4000"]
    bearer_token: "sk-YOUR_MASTER_KEY"
```

Replace `sk-YOUR_MASTER_KEY` with the value of `LITELLM_MASTER_KEY` from your `.env` file. Prometheus uses this token to authenticate its scrape requests against the protected `/metrics` endpoint.

The deployment stack runs LiteLLM behind Traefik, which handles TLS termination and automatic certificate provisioning through Let's Encrypt. PostgreSQL provides persistent storage for virtual keys, spend data, and usage logs. Prometheus collects gateway metrics by scraping the LiteLLM `/metrics` endpoint every 15 seconds over the internal Docker network.

**1. Add your user account to the Docker group:**

``` bash
$ sudo usermod -aG docker $USER
```

**2. Apply the new group membership:**

``` bash
$ newgrp docker
```

**3. Create the Docker Compose manifest file:**

``` bash
$ nano docker-compose.yaml
```

**4. Add the following contents:**

```
services:
  traefik:
    image: traefik:v3.6
    container_name: traefik
    restart: always
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.websecure.address=:443"
      - "--entrypoints.web.http.redirections.entrypoint.to=websecure"
      - "--entrypoints.web.http.redirections.entrypoint.scheme=https"
      - "--certificatesresolvers.le.acme.httpchallenge=true"
      - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web"
      - "--certificatesresolvers.le.acme.email=${LETSENCRYPT_EMAIL}"
      - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./letsencrypt:/letsencrypt

  litellm:
    image: docker.litellm.ai/berriai/litellm:v1.89.1
    container_name: litellm
    restart: always
    volumes:
      - ./config.yaml:/app/config.yaml
    command:
      - "--config=/app/config.yaml"
    environment:
      DATABASE_URL: ${DATABASE_URL}
      STORE_MODEL_IN_DB: "True"
    env_file:
      - .env
    expose:
      - "4000"
    depends_on:
      db:
        condition: service_healthy
    healthcheck:
      test:
        - CMD-SHELL
        - python3 -c "import urllib.request; urllib.request.urlopen('http://localhost:4000/health/liveliness')"
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.litellm.rule=Host(`${DOMAIN}`)"
      - "traefik.http.routers.litellm.entrypoints=websecure"
      - "traefik.http.routers.litellm.tls=true"
      - "traefik.http.routers.litellm.tls.certresolver=le"
      - "traefik.http.services.litellm.loadbalancer.server.port=4000"

  db:
    image: postgres:16
    container_name: litellm_db
    restart: always
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: llmproxy
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - ./postgres:/var/lib/postgresql/data
    expose:
      - "5432"
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 1s
      timeout: 5s
      retries: 10

  prometheus:
    image: prom/prometheus
    container_name: litellm_prometheus
    restart: always
    volumes:
      - ./prometheus:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    expose:
      - "9090"
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.path=/prometheus"
      - "--storage.tsdb.retention.time=15d"
```

This configuration deploys four services behind a single HTTPS endpoint: `traefik` acts as the reverse proxy and TLS terminator, redirecting HTTP to HTTPS via Let's Encrypt; `litellm` runs the proxy image pinned to a specific release tag, mounts `config.yaml`, and waits for PostgreSQL to become healthy before starting; `db` runs PostgreSQL 16 as the persistent backend for virtual keys, spend data, and usage logs; `prometheus` scrapes LiteLLM metrics every 15 seconds with a 15-day retention window. The named volumes for `db` and `prometheus` ensure their data survives container removal or recreation.

LiteLLM Docker images are signed with [Cosign](https://docs.sigstore.dev/quickstart/quickstart-cosign/). Verifying the image signature before deployment confirms the image has not been tampered with since it was published by the LiteLLM team.

**1. Download and install the Cosign binary:**

``` bash
$ curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
```

**2. Move the binary into your system path:**

``` bash
$ sudo mv cosign-linux-amd64 /usr/local/bin/cosign
```

**3. Make the binary executable:**

``` bash
$ sudo chmod +x /usr/local/bin/cosign
```

**4. Verify the installation:**

``` bash
$ cosign version
```

**5. Verify the LiteLLM image signature using the pinned public key.** This checks the same release tag deployed in the Docker Compose file. LiteLLM publishes signatures for the `ghcr.io` registry specifically, so this confirms the `v1.89.1` release itself, not the `docker.litellm.ai` mirror byte-for-byte.

``` bash
$ cosign verify \
    --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \
    ghcr.io/berriai/litellm:v1.89.1
```

A successful verification outputs a JSON payload confirming the image was signed with the LiteLLM public key.

**1. Start the services:**

``` bash
$ docker compose up -d
```

**2. Verify all containers are running:**

``` bash
$ docker compose ps
```

Verify that all four containers show an `Up` status, with `litellm` and `db` marked `(healthy)` and `traefik` listing ports 80 and 443 under `PORTS`.

LiteLLM exposes a web-based admin dashboard at the `/ui` path of your configured domain, with visibility into model configuration, virtual keys, usage, and spend tracking.

`https://litellm.example.com/ui`, replacing `litellm.example.com` with your configured domain.`.env` file — the value of `UI_USERNAME` in the `UI_PASSWORD` in the `config.yaml` appears in the list with its alias and underlying provider model. The left sidebar also provides access to LiteLLM exposes an OpenAI-compatible API, so any application built for the OpenAI SDK works with LiteLLM by pointing `base_url` at the gateway. Virtual keys provide scoped, credential-isolated access without exposing the master key.

**1. Install the Python virtual environment package:**

``` bash
$ sudo apt install -y python3-venv
```

**2. Create a Python virtual environment:**

``` bash
$ python3 -m venv litellm-env
```

**3. Activate the virtual environment:**

``` bash
$ source litellm-env/bin/activate
```

**4. Install the OpenAI Python SDK:**

``` bash
$ pip install openai
```

**5. Export your master key as an environment variable, replacing `sk-YOUR_MASTER_KEY` with the `LITELLM_MASTER_KEY` value from your `.env` file:**

``` bash
$ export LITELLM_MASTER_KEY="sk-YOUR_MASTER_KEY"
```

**6. Export your gateway domain as an environment variable, replacing `litellm.example.com` with your configured domain:**

``` bash
$ export LITELLM_DOMAIN="litellm.example.com"
```

**7. Create the test script:**

``` bash
$ nano test_litellm.py
```

**8. Add the following contents, replacing `my-model` with the model name you set in `config.yaml`:**

``` python
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["LITELLM_MASTER_KEY"],
    base_url=f"https://{os.environ['LITELLM_DOMAIN']}",
)

response = client.chat.completions.create(
    model="my-model",
    messages=[{"role": "user", "content": "What is an AI gateway?"}],
)

print(response.choices[0].message.content)
```

**9. Run the script:**

``` bash
$ python3 test_litellm.py
```

The script returns a response from the configured LLM provider, confirming the gateway is routing requests correctly.

**10. Create a virtual key using the LiteLLM API, replacing `my-model` with the model name you set in `config.yaml`:**

``` bash
$ curl -X POST "https://${LITELLM_DOMAIN}/key/generate" \
    -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
    -H "Content-Type: application/json" \
    -d '{"key_alias": "test-app-key", "models": ["my-model"], "max_budget": 5.00}' \
    | python3 -c "import sys, json; print(json.load(sys.stdin)['key'])"
```

The command prints the generated virtual key, similar to `sk-M9M0_aUDLi7AuhXtOAw6uw`. Each key can have its own model access list, budget limit, and rate limit.

**11. Export the virtual key returned in the previous step, replacing `sk-your-virtual-key` with the full key string printed:**

``` bash
$ export LITELLM_VIRTUAL_KEY="sk-your-virtual-key"
```

**12. Send a request using the virtual key, replacing `my-model` with the model name you set in `config.yaml`:**

``` bash
$ curl -X POST "https://${LITELLM_DOMAIN}/v1/chat/completions" \
    -H "Authorization: Bearer ${LITELLM_VIRTUAL_KEY}" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "my-model",
      "messages": [{"role": "user", "content": "Hello from a virtual key!"}]
    }'
```

The gateway authenticates the virtual key, routes the request to the configured provider, and returns the model response.

**13. Return to the dashboard at `https://litellm.example.com/ui` and click Logs in the left sidebar.** Each entry shows the model alias, key alias, token count, and estimated cost for requests sent through both the master key and the virtual key.

For the full guide with additional tips, visit the original article on **[Vultr Docs](https://docs.vultr.com/how-to-deploy-litellm-open-source-ai-gateway)**.