Gate your app on a Postgres migration job with service_completed_successfully

Originally published on the Hubfly blog.

TL;DR

What you get: a 3-service Compose file (Postgres, a one-shot migration job, an API) where the API cannot start until the migration exits 0. Who it's for: anyone whose app has ever booted against a database with a missing table. Time: about 20 minutes. Our second rollout took 59 seconds from click to "Stack deployment finished". The twist: our first deploy failed. The migration job died in 2 seconds. The likely cause: a brand-new container couldn't resolve the database's hostname yet. The fix is a wait loop, and it's in the complete file at the end of Step 3.

By Octave Nkurunziza · Drafted with AI, tested on Hubfly on 16 October 2026

You push a release. The API boots, runs its first query, and gets relation "todos" does not exist. The migration ran too, just 3 seconds later.

Most Compose files only say depends_on: [db]. That waits for the database container to start, not for the schema.

Docker's docs say so plainly: Compose waits until a dependency is running, not until it is ready (Docker docs).

The fix is well known: a healthcheck on Postgres, a one-shot migration service, and condition: service_completed_successfully on the app. What the popular guides skip is what happens when the gate itself misfires. We ran this on real infrastructure and it misfired on the first try. This post shows the file that survived.

db (postgres:16-alpine)          stage 1
  │  healthcheck: pg_isready over TCP
  ▼  condition: service_healthy
migrate (one-shot job, exits 0)  stage 2
  │  waits for DNS + port, then runs idempotent SQL
  ▼  condition: service_completed_successfully
api (PostgREST, serves /todos)   stage 3

The API is PostgREST, which turns a Postgres schema into a REST API. It's a good test subject because it reads the schema at boot. If the migration hasn't run, there is nothing to serve.

Hubfly's composer turned these 3 services into 3 deploy stages. It marked migrate as a Job with restart policy Never, because of restart: "no".

db:
  image: postgres:16-alpine
  environment:
    POSTGRES_DB: app
    POSTGRES_USER: app
    POSTGRES_PASSWORD: lab-only-change-me   # replace before you deploy
  volumes:
    - pgdata:/var/lib/postgresql/data
  healthcheck:
    test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
    interval: 5s
    timeout: 3s
    retries: 10

Note the -h 127.0.0.1. Without it, pg_isready checks the Unix socket.

On first boot, the official image starts a temporary server to create your database. In our logs, that temporary server listened only on the Unix socket. The real server came up afterwards with listening on IPv4 address "0.0.0.0", port 5432.

A socket check could go green during that setup window, so we switched to TCP as a precaution. A TCP check only passes once other containers can actually connect.

migrate:
  image: postgres:16-alpine
  restart: "no"
  environment:
    PGHOST: "{{service:db.host}}"
    PGUSER: app
    PGPASSWORD: lab-only-change-me
    PGDATABASE: app
  command:
    - sh
    - -c
    - |
      timeout 60 sh -c 'until pg_isready -t 2; do echo "waiting for db"; sleep 2; done' || exit 1
      psql -v ON_ERROR_STOP=1 \
        -c "CREATE TABLE IF NOT EXISTS todos (id serial PRIMARY KEY, task text NOT NULL, done boolean NOT NULL DEFAULT false)" \
        -c "DO 'BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = ''web_anon'') THEN CREATE ROLE web_anon NOLOGIN; END IF; END'" \
        -c "GRANT USAGE ON SCHEMA public TO web_anon" \
        -c "GRANT SELECT ON todos TO web_anon" \
        -c "INSERT INTO todos (task) SELECT 'ship the migration gate' WHERE NOT EXISTS (SELECT 1 FROM todos)"
  depends_on:
    db:
      condition: service_healthy

Three details matter here.

timeout 60 keeps a broken database from hanging the rollout forever; after a minute the job exits 1 and the gate stays closed.ON_ERROR_STOP=1. psql happily continues past a failed statement and exits 0. Your gate would open on a half-migrated schema.NOTICE: relation "todos" already exists, skipping and INSERT 0 0, and exited 0. {{service:db.host}} is Hubfly's service reference. At deploy time it became the database's private hostname, db-1-7befe5b4.eucentral.hubfly.internal.

On plain Docker Compose (we didn't test this part), replace {{service:db.host}} with the service name db in both places: PGHOST: db and PGRST_DB_URI: postgres://app:…@db:5432/app.

The DO '...' block uses ordinary quotes instead of $$. That's deliberate: Compose treats $ as variable interpolation, and quoted SQL dodges the whole escaping question.

  api:
    image: postgrest/postgrest:v12.2.3
    environment:
      PGRST_DB_URI: "postgres://app:lab-only-change-me@{{service:db.host}}:5432/app"
      PGRST_DB_SCHEMAS: public
      PGRST_DB_ANON_ROLE: web_anon
      PGRST_SERVER_PORT: "3000"
    ports:
      - "3000:3000"
    depends_on:
      migrate:
        condition: service_completed_successfully

volumes:
  pgdata:

service_completed_successfully means exit code 0, nothing else. If the job exits 1, the API is never created. That's the whole point: a bad migration stops the release instead of shipping it.

The anonymous role gets SELECT only. We checked: a POST /todos from the browser returned 401 with permission denied for table todos.

For production, have PostgREST log in as a dedicated low-privilege authenticator role instead of the app owner. We kept one user here to keep the file short.

Here's the whole file, exactly as we deployed it:

services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: app
      POSTGRES_USER: app
      POSTGRES_PASSWORD: lab-only-change-me
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U app -d app"]
      interval: 5s
      timeout: 3s
      retries: 10

  migrate:
    image: postgres:16-alpine
    restart: "no"
    environment:
      PGHOST: "{{service:db.host}}"
      PGUSER: app
      PGPASSWORD: lab-only-change-me
      PGDATABASE: app
    command:
      - sh
      - -c
      - |
        timeout 60 sh -c 'until pg_isready -t 2; do echo "waiting for db"; sleep 2; done' || exit 1
        psql -v ON_ERROR_STOP=1 \
          -c "CREATE TABLE IF NOT EXISTS todos (id serial PRIMARY KEY, task text NOT NULL, done boolean NOT NULL DEFAULT false)" \
          -c "DO 'BEGIN IF NOT EXISTS (SELECT FROM pg_roles WHERE rolname = ''web_anon'') THEN CREATE ROLE web_anon NOLOGIN; END IF; END'" \
          -c "GRANT USAGE ON SCHEMA public TO web_anon" \
          -c "GRANT SELECT ON todos TO web_anon" \
          -c "INSERT INTO todos (task) SELECT 'ship the migration gate' WHERE NOT EXISTS (SELECT 1 FROM todos)"
    depends_on:
      db:
        condition: service_healthy

  api:
    image: postgrest/postgrest:v12.2.3
    environment:
      PGRST_DB_URI: "postgres://app:lab-only-change-me@{{service:db.host}}:5432/app"
      PGRST_DB_SCHEMAS: public
      PGRST_DB_ANON_ROLE: web_anon
      PGRST_SERVER_PORT: "3000"
    ports:
      - "3000:3000"
    depends_on:
      migrate:
        condition: service_completed_successfully

volumes:
  pgdata:

pgdata volume, and 2 DEPENDS arrows. The composer's monthly estimate for this stack on the shared tier was api container, open https://<your-subdomain>.eu1.hubfly.app/todos. We got this back:

[{"id":1,"task":"ship the migration gate","done":false}]

A second GET /todos from the browser took 271 ms end to end.

One surprise: ports: "3000:3000" also created a public TCP endpoint on tcp.eu1.hubfly.app by itself. That's fine for an API. Never add ports: to db.

Hubfly marked db healthy about 2 seconds after it deployed, in both runs. So don't rely on the healthcheck alone. The wait loop is what made the second run work.

Our first file didn't have the wait loop or the TCP healthcheck. Here's what happened.

db went healthy at 06:28:24. migrate deployed at 06:28:36 and failed 2 seconds later. The API was never created, so the gate did its job. But why did a migration fail against a healthy database?

The job's container was rolled back, so its logs were gone. We can't prove the cause, but 2 clues from the second run point to it.

The migration job's own first line, with the wait loop in place:

db-1-7befe5b4.eucentral.hubfly.internal:5432 - no response
waiting for db
db-1-7befe5b4.eucentral.hubfly.internal:5432 - accepting connections
CREATE TABLE
...

And PostgREST's first 2 seconds of life (container logs are in UTC; the deploy table is CAT, UTC+2):

04:33:10 Failed to query the PostgreSQL version ... could not translate host name
         "db-1-7befe5b4.eucentral.hubfly.internal" to address: Name does not resolve
04:33:12 Successfully connected to PostgreSQL 16.15
04:33:14 Schema cache loaded 1 Relations

In the second run, a freshly started container could not resolve the database's private hostname for about 2 seconds. PostgREST shrugs that off because it retries. A one-shot psql doesn't retry. It fails, exits non-zero, and takes your deploy down with it.

That's not a Hubfly quirk you can ignore elsewhere. On any orchestrator, "the dependency is healthy" and "my container can reach it right now" are two different facts. A job that runs once must check the second one itself.

1. The job fails in 2 seconds and you can't see why.

A failed job may be rolled back with its logs. Add the wait loop from Step 2, and make the job print what it's doing (echo "waiting for db"). If it still fails, the next run's logs will show whether it's DNS, the port or the SQL.

2. "Healthy" doesn't mean "reachable".

pg_isready without -h checks the Unix socket, and the first-boot setup server listens only there. Use pg_isready -h 127.0.0.1 -U app -d app, and still keep the wait loop in the job: in our runs, db was marked healthy about 2 seconds after deploy either way.

3. A failed rollout leaves an old database behind.

After our failed first deploy, the rollback of db itself failed (Rollback failed for 'db'). The old db was left behind, asleep and still attached to the volume. Our second deploy then created db-1 next to it, on the same pgdata volume.

Two Postgres servers on one data directory is how you lose data. Before you redeploy after a failure, open the project, and stop or remove any leftover database container.

4. The migration isn't idempotent.

The job runs on every deploy. CREATE TABLE without IF NOT EXISTS will fail on deploy number 2 and block every release after it. Guard every statement, or use a migration tool that records what it applied (Prisma's migrate deploy, Flyway, golang-migrate, Alembic).

service_completed_successfully is a good gate. It only works if the job behind it is honest: it waits for the database itself, stops on the first error, and is safe to run again. Put those 3 properties in your migration job and a broken schema stops the release instead of reaching users.

Paste the Compose file into a new Hubfly project (Create → Compose Stack → YAML) and click Deploy Stack. The shared tier needs no card. Change lab-only-change-me first.

The composer estimated $0.01/month for this stack on the shared tier. That matches a 1 GB volume: 1 GB × $0.00001536 per GB-hour × 730 h = $0.011 (the free tier includes 1 GB of storage).

Shared containers auto-sleep, and their compute is free within the free tier's 360 hours a month. Two containers kept awake all month would use 1,460 hours.

If you keep db and api awake using a dedicated container (0.5 vCPU, 256 MB):

CPU: 0.5 = $1.46

RAM: 256 MB = $0.36

Per container: $1.94/month; for db + api: $3.88/month, plus $0.11 of storage

The migration job runs for a couple of seconds per deploy, which rounds to $0.00.

Read next:

{{service:...}} references