{"slug": "how-to-upgrade-postgresql-without-downtime-using-pgbouncer", "title": "How to Upgrade PostgreSQL Without Downtime Using PgBouncer", "summary": "DocSpring released db-cutover, an open-source Python CLI that upgrades PostgreSQL databases with no downtime by running a second database on the target version, copying data via logical replication, and using PgBouncer to shift application connections once the new database has caught up. The tool, rebuilt from an earlier TUI so that AI coding agents can operate it with a human operator, targets PostgreSQL 16 or newer on Amazon RDS with PgBouncer running in Kubernetes in transaction-pooling mode, and requires `--execute` on any command that makes changes. DocSpring says the approach keeps the application online through the copy, with only the final connection switch requiring a pause that must fit within the application's request, proxy, job and database timeouts.", "body_md": "This guide shows how to upgrade PostgreSQL while your application keeps serving requests. We'll create a second database running the target version, copy the data with logical replication, and use PgBouncer to move application connections across once the new database has caught up.\n\nWe built [db-cutover](https://github.com/DocSpring/db-cutover?ref=docspring.com) to run this process at DocSpring. The repository contains a Python CLI, configuration examples, verification tools, and a runbook that a coding agent can follow with a human operator. Each command handles one step, and commands that make changes require `--execute`. This tool was originally a TUI that we would operate ourselves, but we rebuilt it for AI agents to use since we now do almost all of our work using coding agents.\n\nThe goal is to keep the application online throughout the copy and avoid failed requests during the final switch. PgBouncer briefly queues queries while transactions finish and connections move to the new database. That pause must fit within your application's request, proxy, job, and database timeouts. You'll measure it in a rehearsal before switching.\n\nThe existing database is **BLUE**. The new database is **GREEN**:\n\n``` php\nApplication -> PgBouncer -> BLUE: current PostgreSQL version\n                              |\n                       logical replication\n                              |\n                           GREEN: target PostgreSQL version\n```\n\nMost of the migration happens while BLUE continues serving the application. Only the final connection switch needs a pause.\n\n## 1. Check That Your Deployment Fits\n\nThe CLI currently targets PostgreSQL 16 or newer on Amazon RDS, with PgBouncer running in Kubernetes in transaction-pooling mode. The same general approach can work elsewhere, but the AWS and Kubernetes commands in this guide are specific to that setup.\n\nBefore starting, confirm that:\n\n- All foreground database traffic goes through the PgBouncer pods you will switch. These should be dedicated to this workflow: the tool pauses every database on the selected pods.\n- You can stop every client that bypasses PgBouncer, including background workers, schedulers, consoles, and migration processes, and prevent them from reconnecting during the switch.\n- PgBouncer reads its configuration from a directory-mounted ConfigMap, without automatic reloads. You can control pod rollouts and restarts during the switch.\n- Your application and extensions work on the target PostgreSQL version, and you have a tested backup and recovery process.\n\nFreeze schema migrations from the schema copy until replication cleanup. PostgreSQL logical replication does not carry DDL or sequence counters to the destination. The tool handles sequences separately and blocks objects it does not migrate, including large objects, unlogged tables, foreign tables, and materialized views. Review the [PostgreSQL replication restrictions](https://www.postgresql.org/docs/17/logical-replication-restrictions.html?ref=docspring.com) for your schema.\n\n## 2. Install the Tool and Create a Private Config\n\nYou'll need Python 3.11+, [uv](https://docs.astral.sh/uv/?ref=docspring.com), the AWS and Kubernetes CLIs, and current `pg_dump` and `psql` clients at least as new as both databases. Use matching client versions. The operator's machine must reach both databases and the Kubernetes API; GREEN must also reach BLUE directly for replication.\n\n```\ngit clone https://github.com/DocSpring/db-cutover.git\ncd db-cutover\ngit checkout c32812f4d657040f4330563f45ffcf8e75df351c\nuv sync --frozen\nuv run db-cutover --help\n\nmkdir -p \"$HOME/db-cutover-run\"\ncp examples/config.toml \"$HOME/db-cutover-run/cutover.toml\"\nexport C=\"$HOME/db-cutover-run/cutover.toml\"\n```\n\nKeep that checkout fixed throughout the migration. Edit the copied config with your database name, AWS region/profile, instance identifiers, Kubernetes context, and direct-writer deployments. The [complete example](https://github.com/DocSpring/db-cutover/blob/c32812f4d657040f4330563f45ffcf8e75df351c/examples/config.toml?ref=docspring.com) documents the settings. For example:\n\n```\nname = \"staging-upgrade\"\ndatabase = \"app\"\nadmin_user = \"postgres\"\napp_user = \"app\"\n\n[aws]\nregion = \"us-east-1\"\n\n[blue]\nidentifier = \"example-blue\"\n\n[green]\nidentifier = \"example-green\"\n\n[secrets]\nadmin_password = \"env:CUTOVER_ADMIN_PASSWORD\"\napp_password = \"env:CUTOVER_APP_PASSWORD\"\npgbouncer_admin_password = \"env:CUTOVER_PGBOUNCER_PASSWORD\"\n\n[pgbouncer]\nkube_context = \"example-staging-cluster\"\nhost = \"pgbouncer.example.com\"\n```\n\nThese are placeholders, not a ready-to-run deployment. Supply credentials through your secret manager or environment; don't paste them into the config or commit the operational files. The CLI also supports 1Password references. Its schema dumps and reports belong in the private working directory.\n\nCheck the environment:\n\n```\nuv run db-cutover \"$C\" env\nuv run db-cutover \"$C\" preflight\n```\n\nSome checks will fail until you complete the preparation below. Read each failure and resolve it before the step that depends on it. A dry run can still connect to databases and run expensive reads; it means the command won't make the remote change.\n\n## 3. Prepare BLUE for Logical Replication\n\nYou'll need to enable logical replication on BLUE, so unfortunately this may require some downtime during your first database migration. It's important to keep `wal_level = logical` enabled on GREEN so that you won't need to reboot again in a future migration.\n\nIf BLUE already has `wal_level = logical`, skip the reboot. Otherwise, enable `rds.logical_replication` in its RDS parameter group and apply the change. This is a static parameter, so RDS requires a reboot. The [AWS instructions](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL.Concepts.General.FeatureSupport.LogicalReplication.html?ref=docspring.com) describe the parameter and role requirements.\n\n```\nuv run db-cutover \"$C\" reboot-blue\nuv run db-cutover \"$C\" reboot-blue --execute\n```\n\nThe command pauses PgBouncer while BLUE restarts. Stop direct clients first. A reboot can last much longer than the eventual connection switch; if that wait exceeds your timeout budget, arrange this prerequisite separately. It cannot be made invisible merely by putting a pool in front of it.\n\nNext, prepare the source tables:\n\n```\nuv run db-cutover \"$C\" prepare-blue\nuv run db-cutover \"$C\" prepare-blue --execute\n```\n\nThis adds `REPLICA IDENTITY FULL` where a primary key is missing and creates a table for synthetic probe and marker rows. FULL identity has performance and data-type limitations, so test those tables rather than assuming replication will be cheap.\n\nStart the monitors in separate terminals:\n\n```\nuv run db-cutover \"$C\" watch\nuv run db-cutover \"$C\" probe --execute\n```\n\n`watch` reports changes in application health, replication and PgBouncer state. If your trusted HTTPS health endpoints require a token, configure `[secrets] health_check_token`; the tool sends it as `X-Health-Check-Token` and refuses redirects. `probe` writes through PgBouncer as the application user and reports errors, slow writes and changes of database. Check that both processes stay alive.\n\n## 4. Provision GREEN and Copy the Schema\n\nChoose an RDS engine version available in your region and test your application against it. Set `TARGET_VERSION` to that exact version. If your Terraform layout matches the repository's example, set `[terraform] dir` and the resource names to match your files, then generate a draft configuration:\n\n```\nuv run db-cutover \"$C\" green-tf --engine-version \"$TARGET_VERSION\"\nuv run db-cutover \"$C\" green-tf --engine-version \"$TARGET_VERSION\" --execute\n```\n\nReview the generated file and a Terraform plan before applying it. Check instance parameters that changed between PostgreSQL versions, availability, storage, encryption, backups, and network access. The plan should create GREEN without replacing or destroying BLUE. Provisioning with your own infrastructure tools is also fine; the Terraform helpers are optional.\n\nUpdate `[green] identifier` to the provisioned instance, or use the example's Terraform-derived identifier setting. GREEN starts empty. Install any required extension packages and create roles referenced by the schema, then copy it:\n\n```\nuv run db-cutover \"$C\" copy-schema\nuv run db-cutover \"$C\" copy-schema --execute\n```\n\nThe tool creates the destination database when needed and restores a schema-only dump. It excludes ownership, grants, publications and subscriptions. Recreate application roles and permissions through your normal provisioning process. Other databases and instance-level settings need their own migration plan.\n\nKeep application writers off GREEN while replication is running.\n\n## 5. Copy the Data and Keep It Current\n\n```\nuv run db-cutover \"$C\" start-replication\nuv run db-cutover \"$C\" start-replication --execute\nuv run db-cutover \"$C\" wait-sync --execute\n```\n\nThe publication on BLUE and subscription on GREEN perform the initial copy, then continue streaming changes. BLUE stays in service throughout this phase. The tool defaults to text replication because [binary replication is less portable across PostgreSQL versions and types](https://www.postgresql.org/docs/17/sql-createsubscription.html?ref=docspring.com).\n\n`wait-sync --execute` waits for every table to finish copying, writes a unique marker on BLUE, and waits for that marker to appear on GREEN. This establishes progress through the configured replication stream. It doesn't prove that an omitted table or a separate subscription is current, so the cutover also checks publication coverage and subscription configuration.\n\nWatch WAL retention and free disk space on BLUE. A slow or stopped subscriber can cause the replication slot to retain enough WAL to fill the source disk. If errors keep increasing, inspect GREEN's PostgreSQL logs privately and fix the cause before continuing.\n\n## 6. Verify the Copy and Advance Sequences\n\n```\nuv run db-cutover \"$C\" verify\nuv run db-cutover \"$C\" verify --execute\nuv run db-cutover \"$C\" sequences\nuv run db-cutover \"$C\" sequences --execute\n```\n\nVerification compares schemas, privileges, row counts, and sampled row-content hashes. Without `--execute`, it is read-only. With `--execute`, it also writes a replication marker. These are useful checks, not an exhaustive comparison or a shared snapshot of both databases. Concurrent writes can cause temporary mismatches; persistent differences need investigation.\n\nSequences need their own step because their counters don't replicate with the rows. GREEN may contain every existing ID while its next sequence value still points at an ID already in use. The tool advances GREEN's sequences above BLUE's reserved values, using a configurable gap, then checks them again during the switch.\n\nYour application must tolerate gaps in IDs. The gap does not make simultaneous writes to BLUE and GREEN safe: it can't resolve conflicting updates, deletes or other unique constraints.\n\n## 7. Rehearse the Pause\n\nStop and fence the direct writers, leaving PgBouncer's configuration pointed at BLUE. Then run:\n\n```\nuv run db-cutover \"$C\" cutover --execute --rehearse\n```\n\nThis pauses the real pools, checks remaining sessions, waits for a replication marker, and compares IDs and sequence state. It stops before RELOAD and resumes traffic on BLUE.\n\nUse the reported pause and probe output to check your timeout budget. This is a live operation, not a dry run. It also doesn't exercise every part of the real switch: staging tests must cover reloading onto GREEN, partial failures, and application behavior on the new database.\n\n## 8. Prepare PgBouncer and Switch Traffic\n\nKeep direct writers stopped. Prevent PgBouncer rollouts, scaling and automatic reloads. Change the database host in its ConfigMap to GREEN, but **don't reload PgBouncer yet**. Wait for the mounted files to update:\n\n```\nuv run db-cutover \"$C\" pgbouncer config\n```\n\nEvery pod should report **running BLUE, file GREEN**.\n\nThis intermediate state needs care. A pod that restarts now reads GREEN immediately. The tool checks for an unexpected target before switching, but that check cannot prevent a restart afterwards. Keep the window short; if your deployment cannot control it, design an external traffic gate before using this workflow.\n\nRun the preconditions, then execute the switch:\n\n```\nuv run db-cutover \"$C\" cutover\nuv run db-cutover \"$C\" cutover --execute\n```\n\nThe switch performs these steps:\n\n1. Pause every selected PgBouncer pod and let active transactions finish.\n2. Check that other client sessions have left BLUE.\n3. Write a marker on BLUE and wait until GREEN applies it.\n4. Compare maximum integer primary-key values and advance any sequences still too close to BLUE's range.\n5. Reload each pod and confirm that every pod points at GREEN.\n6. Resume queued work and check the resulting connections.\n\nPgBouncer's [administration commands](https://www.pgbouncer.org/usage.html?ref=docspring.com) provide the pause, reload and resume operations. The CLI coordinates them across pods and saves a report in the private state directory.\n\nWatch the probe change to GREEN, check real application operations, and confirm that BLUE has no remaining application connections.\n\nIf a failure happens before RELOAD, the tool attempts to resume BLUE and reports any failure to do so. Once RELOAD starts, mixed or unknown targets remain paused and the command fails. Keep direct writers stopped, inspect every pod, align them all to GREEN, and only then resume. The repository's [recovery procedure](https://github.com/DocSpring/db-cutover/blob/c32812f4d657040f4330563f45ffcf8e75df351c/docs/procedure.md?ref=docspring.com#recovery) covers this state. Don't return traffic to BLUE after GREEN has accepted writes: BLUE doesn't have those new transactions.\n\n## 9. Move Direct Writers and Remove Replication\n\nChange direct connection strings to GREEN, restart those clients, and confirm their sessions reach the new database. Prevent application roles, scheduled tasks and consoles from reconnecting to BLUE. A quiet period in database statistics is useful evidence, but it doesn't stop a future writer.\n\nStop the probe before removing its table, then run:\n\n```\nuv run db-cutover \"$C\" finish\nuv run db-cutover \"$C\" finish --execute\n```\n\nCleanup checks the PgBouncer targets, BLUE sessions and write activity, and performs a final marker check before dropping the subscription, slot, publication and probe table.\n\nIf you used the Terraform helpers, reconcile the resource names:\n\n```\nuv run db-cutover \"$C\" promote-tf\nuv run db-cutover \"$C\" promote-tf --execute\n```\n\nReview and run the printed state-move commands using your normal state backups and locking. A subsequent plan should not create, replace or destroy either database. Update references to the primary and retain BLUE according to your recovery policy before deleting it in a separate change. Keeping BLUE is useful for investigation; it isn't an immediate rollback once GREEN contains new writes.\n\nThe [db-cutover repository](https://github.com/DocSpring/db-cutover?ref=docspring.com) contains the complete runbook, configuration reference, failure-path tests, and disposable PostgreSQL replication tests. Start with a staging migration, give your agent the repository's `AGENTS.md`, and use that run to establish the commands, traffic controls and timeout budget for your own application.", "url": "https://wpnews.pro/news/how-to-upgrade-postgresql-without-downtime-using-pgbouncer", "canonical_source": "https://docspring.com/blog/posts/postgresql-blue-green-cutovers-with-pgbouncer/", "published_at": "2026-10-26 22:56:11+00:00", "updated_at": "2026-10-26 23:42:06.177544+00:00", "lang": "en", "topics": ["developer-tools", "ai-agents", "ai-tools"], "entities": ["DocSpring", "PostgreSQL", "PgBouncer", "Amazon RDS", "Kubernetes", "db-cutover"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-upgrade-postgresql-without-downtime-using-pgbouncer", "markdown": "https://wpnews.pro/news/how-to-upgrade-postgresql-without-downtime-using-pgbouncer.md", "text": "https://wpnews.pro/news/how-to-upgrade-postgresql-without-downtime-using-pgbouncer.txt", "jsonld": "https://wpnews.pro/news/how-to-upgrade-postgresql-without-downtime-using-pgbouncer.jsonld"}}