Day 7: Docker Compose — Architecting a Multi-Container Database and Inference Stack Locally

A production-ready RAG platform is not one script. It needs several specialised services on a shared internal network. Docker Compose is the declarative, infrastructure-as-code way to assemble them on a single host, and it closes Week 1 by pulling together everything so far.

Diagram: FIG 1 · A gateway publishes one port. The vector store and inference engine sit on a private bridge network, each with its own persistent volume. See the animated version.

Three services make up the stack:

name: ai-stack

networks:
  ai-internal-net:
    driver: bridge

volumes:
  qdrant_storage:
  ollama_storage:

services:
  vector-store:
    image: qdrant/qdrant:v1.9.0
    networks: [ai-internal-net]
    volumes:
      - qdrant_storage:/qdrant/storage
    healthcheck:
      test: ["CMD-SHELL", "bash -c ':> /dev/tcp/127.0.0.1/6333' || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 10

  inference-engine:
    image: ollama/ollama:latest          # pin a tested tag in production
    networks: [ai-internal-net]
    volumes:
      - ollama_storage:/root/.ollama
    deploy:
      resources:
        limits:
          memory: 10G                    # leave about 6 GB of a 16 GB host free
    healthcheck:
      test: ["CMD", "ollama", "list"]
      interval: 15s
      timeout: 10s
      retries: 10

  ai-gateway:
    build: ./gateway
    networks: [ai-internal-net]
    ports:
      - "80:8000"                        # the only port published to the host
    environment:
      VECTOR_DB_HOST: vector-store
      VECTOR_DB_PORT: "6333"
      LLM_HOST: inference-engine
      LLM_PORT: "11434"
    depends_on:
      vector-store:
        condition: service_healthy
      inference-engine:
        condition: service_healthy

Start it with docker compose up -d --build, check it with docker compose ps, and stop it with docker compose down. Add -v to down only when you want the volumes deleted too.

1. Service discovery by name. Inside a Compose network every service name is a DNS name. Use it, and never hard-code IP addresses or localhost: container IPs change on restart, and localhost points at the calling container itself.

Diagram: FIG 2 · Inside a Compose network, each service name is a DNS name. Hard-coded IPs break on restart, and localhost points at the calling container itself. See the animated version.

2. Healthchecks and dependency ordering. depends_on with condition: service_healthy makes the gateway wait until both services actually pass their checks, so it never queries a database or model server that is still starting.

Diagram: FIG 3 · The gateway waits until the vector store and the inference engine pass their healthchecks, so it never queries a service that is still starting. See the animated version.

3. Volume decoupling. Keep persistent data in named volumes, outside the containers. You can then upgrade, restart or rebuild the stack without losing the vector index or re-down model weights.

Diagram: FIG 4 · Persistent data lives in named volumes, so containers stay disposable. A memory limit on the inference engine protects the host. See the animated version.

Two practical notes. The memory limit under deploy.resources is honoured by current docker compose, but check it with docker stats on your setup. And the healthcheck commands depend on what each image ships, so confirm they work with docker compose ps showing healthy.

That completes Week 1: lean images, GPU access, quantized weights, dedicated runtimes, a validated API and a stack that starts in the right order. Next week moves it all onto Kubernetes, starting with a lightweight cluster on a 16 GB machine.

This is the exact example from the video. Docker Compose itself reads this file, and a short check prints what it found. Nothing is started, and the gateway image is a placeholder.

name: ai-stack
services:
  vector-store:
    image: qdrant/qdrant:v1.9.0
    volumes: [qdrant_data:/qdrant/storage]
  inference-engine:
    image: ollama/ollama:latest
    volumes: [ollama_data:/root/.ollama]
  ai-gateway:
    image: ai-gateway:dev
    ports: ["80:8000"]
    environment:
      VECTOR_DB_HOST: vector-store
      LLM_HOST: inference-engine
    depends_on: [vector-store, inference-engine]
volumes: {qdrant_data: {}, ollama_data: {}}

Helper file check.py, used by the run:

import json, subprocess
raw = subprocess.check_output(["docker", "compose", "config", "--format", "json"])
c = json.loads(raw)
svc = c["services"]
print("services:", len(svc))
for n, s in svc.items():
    for p in s.get("ports", []):
        print("published:", n, p["published"] + "->" + str(p["target"]))
g = svc["ai-gateway"]
print("db host:", g["environment"]["VECTOR_DB_HOST"])
for d in sorted(g["depends_on"]):
    print("waits for:", d)
print("volumes:", ", ".join(sorted(c["volumes"])))

Run it:

python check.py

Output I got while recording the video:

services: 3
published: ai-gateway 80->8000
db host: vector-store
waits for: inference-engine
waits for: vector-store
volumes: ollama_data, qdrant_data

Day 8: K3d and lightweight Kubernetes: running multi-node clusters on 16 GB of RAM.

Originally published at https://sureshpallapothu.in/blog/ai-int-day-7-docker-compose, where this post includes animated diagrams.