Deploying LiteLLM: An Open-Source AI Gateway

A developer published a deployment guide for LiteLLM, an open-source AI gateway that exposes a unified, OpenAI-compatible API across more than 100 large language model providers. The walkthrough covers running LiteLLM on a Linux server via Docker Compose with PostgreSQL for persistent storage, Prometheus for metrics, and Traefik as an HTTPS reverse proxy, including virtual key issuance, budget limits, spend tracking, and centralized routing.

LiteLLM https://litellm.ai is an open-source AI gateway that provides a unified, OpenAI-compatible API for over 100 large language model LLM providers, removing the need to integrate with each provider's SDK and authentication scheme. In production, teams use LiteLLM to issue virtual keys with budget limits, track token usage and spend across providers, and configure centralized access control and routing. This guide walks through deploying LiteLLM on a Linux server using Docker Compose with PostgreSQL for persistent storage, Prometheus for metrics collection, and Traefik as the reverse proxy for HTTPS access. By the end, you'll have a fully functional, OpenAI-compatible AI gateway accessible over a custom domain, with virtual key management, spend tracking, and provider routing configured. Before you begin, you need a Linux-based server with at least 4 CPU cores and 8 GB of RAM as a non-root user with sudo privileges, Docker and Docker Compose installed, a DNS A record such as litellm.example.com pointing to your server's IP address, and an API key from at least one supported LLM provider https://docs.litellm.ai/docs/providers . LiteLLM requires a configuration file to define model providers and routing rules, a Prometheus configuration file for metrics scraping, and environment variables for secrets and database credentials. 1. Create the project directory with subdirectories for persistent data: bash $ mkdir -p ~/litellm/{letsencrypt,postgres,prometheus} letsencrypt stores SSL/TLS certificates, postgres persists PostgreSQL database files, and prometheus persists Prometheus metrics data. 2. Navigate to the project directory: bash $ cd ~/litellm 3. Set matching ownership on the Prometheus host directory Prometheus runs as UID 65534 inside the container : bash $ sudo chown -R 65534:65534 prometheus 4. Generate a master key and salt key for LiteLLM. Run this command twice to produce two separate values: bash $ openssl rand -hex 32 Save both values for the LITELLM MASTER KEY and LITELLM SALT KEY fields in the next step. 5. Create the .env file to store credentials and secrets: bash $ nano .env 6. Add the following values: DOMAIN=litellm.example.com LETSENCRYPT EMAIL=admin@example.com LITELLM MASTER KEY=sk-YOUR MASTER KEY LITELLM SALT KEY=sk-YOUR SALT KEY LLM PROVIDER API KEY=YOUR LLM PROVIDER API KEY POSTGRES PASSWORD=STRONG DATABASE PASSWORD DATABASE URL=postgresql://llmproxy:${POSTGRES PASSWORD}@db:5432/litellm UI USERNAME=admin UI PASSWORD=YOUR UI PASSWORD Replace the placeholders with your own values: litellm.example.com is the domain pointing to your server's IP address; admin@example.com is the email address for Let's Encrypt notifications; sk-YOUR MASTER KEY is the admin key used to authenticate with the LiteLLM API replace YOUR MASTER KEY with the first generated value and keep the sk- prefix ; sk-YOUR SALT KEY encrypts provider credentials stored in PostgreSQL and cannot be changed after the first model is added; YOUR LLM PROVIDER API KEY is the API key for the provider configured in config.yaml ; STRONG DATABASE PASSWORD is the PostgreSQL password used by both the database container and the connection string; admin / YOUR UI PASSWORD are the LiteLLM dashboard credentials. 7. Create the LiteLLM configuration file: bash $ nano config.yaml 8. Add the following contents: model list: - model name: my-model litellm params: model: provider/model api key: os.environ/LLM PROVIDER API KEY litellm settings: callbacks: - prometheus require auth for metrics endpoint: true general settings: master key: os.environ/LITELLM MASTER KEY Replace my-model with a name of your choice to identify this model within LiteLLM, and provider/model with the LiteLLM provider prefix followed by the model identifier for example, anthropic/claude-haiku-4-5 . The os.environ/ prefix tells LiteLLM to read the value from an environment variable at runtime rather than hardcoding it. The litellm settings block enables Prometheus metrics via the callbacks field, and require auth for metrics endpoint: true restricts the /metrics endpoint to authenticated requests, so only Prometheus sending the master key as a Bearer token can scrape metrics successfully. The general settings block defines the master key LiteLLM uses to authenticate admin API requests and virtual key management operations. 9. Create the Prometheus configuration file: bash $ nano prometheus.yml 10. Add the following contents: global: scrape interval: 15s evaluation interval: 15s scrape configs: - job name: "litellm" static configs: - targets: "litellm:4000" bearer token: "sk-YOUR MASTER KEY" Replace sk-YOUR MASTER KEY with the value of LITELLM MASTER KEY from your .env file. Prometheus uses this token to authenticate its scrape requests against the protected /metrics endpoint. The deployment stack runs LiteLLM behind Traefik, which handles TLS termination and automatic certificate provisioning through Let's Encrypt. PostgreSQL provides persistent storage for virtual keys, spend data, and usage logs. Prometheus collects gateway metrics by scraping the LiteLLM /metrics endpoint every 15 seconds over the internal Docker network. 1. Add your user account to the Docker group: bash $ sudo usermod -aG docker $USER 2. Apply the new group membership: bash $ newgrp docker 3. Create the Docker Compose manifest file: bash $ nano docker-compose.yaml 4. Add the following contents: services: traefik: image: traefik:v3.6 container name: traefik restart: always command: - "--providers.docker=true" - "--providers.docker.exposedbydefault=false" - "--entrypoints.web.address=:80" - "--entrypoints.websecure.address=:443" - "--entrypoints.web.http.redirections.entrypoint.to=websecure" - "--entrypoints.web.http.redirections.entrypoint.scheme=https" - "--certificatesresolvers.le.acme.httpchallenge=true" - "--certificatesresolvers.le.acme.httpchallenge.entrypoint=web" - "--certificatesresolvers.le.acme.email=${LETSENCRYPT EMAIL}" - "--certificatesresolvers.le.acme.storage=/letsencrypt/acme.json" ports: - "80:80" - "443:443" volumes: - /var/run/docker.sock:/var/run/docker.sock:ro - ./letsencrypt:/letsencrypt litellm: image: docker.litellm.ai/berriai/litellm:v1.89.1 container name: litellm restart: always volumes: - ./config.yaml:/app/config.yaml command: - "--config=/app/config.yaml" environment: DATABASE URL: ${DATABASE URL} STORE MODEL IN DB: "True" env file: - .env expose: - "4000" depends on: db: condition: service healthy healthcheck: test: - CMD-SHELL - python3 -c "import urllib.request; urllib.request.urlopen 'http://localhost:4000/health/liveliness' " interval: 30s timeout: 10s retries: 3 start period: 40s labels: - "traefik.enable=true" - "traefik.http.routers.litellm.rule=Host ${DOMAIN} " - "traefik.http.routers.litellm.entrypoints=websecure" - "traefik.http.routers.litellm.tls=true" - "traefik.http.routers.litellm.tls.certresolver=le" - "traefik.http.services.litellm.loadbalancer.server.port=4000" db: image: postgres:16 container name: litellm db restart: always environment: POSTGRES DB: litellm POSTGRES USER: llmproxy POSTGRES PASSWORD: ${POSTGRES PASSWORD} volumes: - ./postgres:/var/lib/postgresql/data expose: - "5432" healthcheck: test: "CMD-SHELL", "pg isready -d litellm -U llmproxy" interval: 1s timeout: 5s retries: 10 prometheus: image: prom/prometheus container name: litellm prometheus restart: always volumes: - ./prometheus:/prometheus - ./prometheus.yml:/etc/prometheus/prometheus.yml expose: - "9090" command: - "--config.file=/etc/prometheus/prometheus.yml" - "--storage.tsdb.path=/prometheus" - "--storage.tsdb.retention.time=15d" This configuration deploys four services behind a single HTTPS endpoint: traefik acts as the reverse proxy and TLS terminator, redirecting HTTP to HTTPS via Let's Encrypt; litellm runs the proxy image pinned to a specific release tag, mounts config.yaml , and waits for PostgreSQL to become healthy before starting; db runs PostgreSQL 16 as the persistent backend for virtual keys, spend data, and usage logs; prometheus scrapes LiteLLM metrics every 15 seconds with a 15-day retention window. The named volumes for db and prometheus ensure their data survives container removal or recreation. LiteLLM Docker images are signed with Cosign https://docs.sigstore.dev/quickstart/quickstart-cosign/ . Verifying the image signature before deployment confirms the image has not been tampered with since it was published by the LiteLLM team. 1. Download and install the Cosign binary: bash $ curl -O -L "https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64" 2. Move the binary into your system path: bash $ sudo mv cosign-linux-amd64 /usr/local/bin/cosign 3. Make the binary executable: bash $ sudo chmod +x /usr/local/bin/cosign 4. Verify the installation: bash $ cosign version 5. Verify the LiteLLM image signature using the pinned public key. This checks the same release tag deployed in the Docker Compose file. LiteLLM publishes signatures for the ghcr.io registry specifically, so this confirms the v1.89.1 release itself, not the docker.litellm.ai mirror byte-for-byte. bash $ cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.1 A successful verification outputs a JSON payload confirming the image was signed with the LiteLLM public key. 1. Start the services: bash $ docker compose up -d 2. Verify all containers are running: bash $ docker compose ps Verify that all four containers show an Up status, with litellm and db marked healthy and traefik listing ports 80 and 443 under PORTS . LiteLLM exposes a web-based admin dashboard at the /ui path of your configured domain, with visibility into model configuration, virtual keys, usage, and spend tracking. https://litellm.example.com/ui , replacing litellm.example.com with your configured domain. .env file — the value of UI USERNAME in the UI PASSWORD in the config.yaml appears in the list with its alias and underlying provider model. The left sidebar also provides access to LiteLLM exposes an OpenAI-compatible API, so any application built for the OpenAI SDK works with LiteLLM by pointing base url at the gateway. Virtual keys provide scoped, credential-isolated access without exposing the master key. 1. Install the Python virtual environment package: bash $ sudo apt install -y python3-venv 2. Create a Python virtual environment: bash $ python3 -m venv litellm-env 3. Activate the virtual environment: bash $ source litellm-env/bin/activate 4. Install the OpenAI Python SDK: bash $ pip install openai 5. Export your master key as an environment variable, replacing sk-YOUR MASTER KEY with the LITELLM MASTER KEY value from your .env file: bash $ export LITELLM MASTER KEY="sk-YOUR MASTER KEY" 6. Export your gateway domain as an environment variable, replacing litellm.example.com with your configured domain: bash $ export LITELLM DOMAIN="litellm.example.com" 7. Create the test script: bash $ nano test litellm.py 8. Add the following contents, replacing my-model with the model name you set in config.yaml : python import os from openai import OpenAI client = OpenAI api key=os.environ "LITELLM MASTER KEY" , base url=f"https://{os.environ 'LITELLM DOMAIN' }", response = client.chat.completions.create model="my-model", messages= {"role": "user", "content": "What is an AI gateway?"} , print response.choices 0 .message.content 9. Run the script: bash $ python3 test litellm.py The script returns a response from the configured LLM provider, confirming the gateway is routing requests correctly. 10. Create a virtual key using the LiteLLM API, replacing my-model with the model name you set in config.yaml : bash $ curl -X POST "https://${LITELLM DOMAIN}/key/generate" \ -H "Authorization: Bearer ${LITELLM MASTER KEY}" \ -H "Content-Type: application/json" \ -d '{"key alias": "test-app-key", "models": "my-model" , "max budget": 5.00}' \ | python3 -c "import sys, json; print json.load sys.stdin 'key' " The command prints the generated virtual key, similar to sk-M9M0 aUDLi7AuhXtOAw6uw . Each key can have its own model access list, budget limit, and rate limit. 11. Export the virtual key returned in the previous step, replacing sk-your-virtual-key with the full key string printed: bash $ export LITELLM VIRTUAL KEY="sk-your-virtual-key" 12. Send a request using the virtual key, replacing my-model with the model name you set in config.yaml : bash $ curl -X POST "https://${LITELLM DOMAIN}/v1/chat/completions" \ -H "Authorization: Bearer ${LITELLM VIRTUAL KEY}" \ -H "Content-Type: application/json" \ -d '{ "model": "my-model", "messages": {"role": "user", "content": "Hello from a virtual key "} }' The gateway authenticates the virtual key, routes the request to the configured provider, and returns the model response. 13. Return to the dashboard at https://litellm.example.com/ui and click Logs in the left sidebar. Each entry shows the model alias, key alias, token count, and estimated cost for requests sent through both the master key and the virtual key. For the full guide with additional tips, visit the original article on Vultr Docs https://docs.vultr.com/how-to-deploy-litellm-open-source-ai-gateway .