cd /news/ai-infrastructure/how-openai-scales-postgresql-to-800-… · home › topics › ai-infrastructure › article
[ARTICLE · art-171687] src=dev.to ↗ pub= topic=ai-infrastructure verified=true sentiment=↑ positive

How OpenAI scales PostgreSQL to 800 million users

OpenAI scales a single PostgreSQL primary, backed by nearly 50 cross-region read replicas, to serve ChatGPT's 800 million users at millions of queries per second, according to the company's January 2026 engineering post. Rather than shard the existing deployment, OpenAI has frozen new tables in Postgres and routes new features to sharded systems like Azure Cosmos DB, while moving write-heavy workloads off the primary. The company attributes most outages to a retry-driven overload loop and counters it with replica reads, lazy writes, rate-limited backfills, transaction timeouts and a hot standby.

recommendation

Set idle_in_transaction_session_timeout (OpenAI uses 60s) and rate-limit backfills so a load spike can't trigger the retry loop that takes the primary down.

more in playbook: Running AI in production
by read10 min views8 publishedNov 18, 2026

OpenAI scales PostgreSQL for ChatGPT with one primary server that takes every write and nearly 50 read replicas spread across regions, and that handles millions of queries per second for 800 million users. Most of the work goes into keeping load off the primary, and into limits at every layer so one spike can't snowball into an outage.

Below is each problem they hit and how they fixed it, checked against OpenAI's own engineering post from January 2026, plus the parts I'd copy in a normal app, with SQL I tested on Postgres 18. The video above covers the same ground.

This came as a shock to me. I had no idea a single Postgres write instance could go that far. The primary is an Azure Database for PostgreSQL flexible server, and OpenAI says its Postgres load grew more than 10x in the past year.

They haven't sharded it, because sharding the existing workloads would mean changing hundreds of application endpoints and could take months or years. Instead they've drawn a line: no new tables go into this Postgres deployment, and new features default to sharded systems like Azure Cosmos DB.

Writes are the weak spot. Postgres uses multiversion concurrency control (MVCC), so updating even one field copies the whole row into a new version. Under heavy writes that means write amplification, dead rows that queries have to skip past, table and index bloat, and autovacuum work. Reads scale out across replicas. Writes all land on one machine.

OpenAI says its Postgres outages usually follow the same loop. Something upstream causes a sudden jump in load. The cache layer fails and every request misses, or a new feature launch floods the database with writes. A burst of expensive multi-table joins eating the CPU can do it too. Queries slow down and start timing out. Clients retry, the retries add even more load, and it keeps getting worse until the whole service is down.

Every fix below breaks that loop somewhere.

Reads go to replicas wherever possible. The only reads left on the primary are the ones inside write transactions, and those get tuned so they stay fast.

For writes, OpenAI moved the shardable, write-heavy workloads to Cosmos DB. The harder-to-shard ones are still being moved. Then they went through their own code. They fixed bugs that wrote the same data twice and added lazy writes to spread spikes out. Backfills got strict rate limits, so filling in a column can't hammer the primary (more on that below).

You can hunt for duplicate writes in a small app the same way. Turn on the pg_stat_statements extension, sort by how often each query runs, and look for writes that run more often than they should.

OpenAI found a query that joined 12 tables, and spikes in it caused some of their worst incidents. Their advice is to avoid multi-table joins where you can, and to break up the ones you need and do the join in application code instead.

They also say many of the bad queries came from ORMs. Prisma, Drizzle, SQLAlchemy and Django's ORM all make it easy to write code that turns into slow SQL without you noticing, so read the SQL your ORM generates for your hottest endpoints.

Then set a timeout for transactions that sit open doing nothing. OpenAI uses idle_in_transaction_session_timeout, because those sessions stop autovacuum from cleaning up dead rows:

ALTER DATABASE postgres SET idle_in_transaction_session_timeout = '60s';

Swap postgres for your database name. New connections pick it up. When I tested it with a 3-second value, a session that ran BEGIN and then sat idle got cut off with FATAL: terminating connection due to idle-in-transaction timeout.

A single writer is a single point of failure. OpenAI runs the primary in high-availability mode with a hot standby, a replica kept in sync that can be promoted quickly if the primary dies or needs maintenance. In the video I called this zero downtime, but the post only promises to keep downtime short.

The bigger fix is that most critical requests only read data, and those reads now go to replicas. If the primary goes down, writes fail but ChatGPT keeps answering reads, so it's no longer their worst severity of incident. They also run more than one replica in every region, so losing one replica doesn't take a region down.

Sometimes one workload, like a new feature with inefficient queries, eats most of the CPU and slows everything else on the same instance. OpenAI splits requests into high-priority and low-priority tiers and routes each tier to its own instances. A heavy low-priority job can then max out its own servers without touching high-priority traffic. They do the same between products, so one product's traffic can't hurt another.

Each Azure PostgreSQL instance allows at most 5,000 connections, and OpenAI has had incidents where a connection storm used them all up. (Plain Postgres has no fixed number. max_connections is a setting, 100 by default.)

Their fix is PgBouncer in statement or transaction pooling mode, so many clients share a few real connections. In their benchmarks the average time to get a connection dropped from 50 ms to 5 ms. PgBouncer runs in the same region as the clients and the replica it sits in front of, and every read replica has its own Kubernetes deployment running several PgBouncer pods behind one Kubernetes Service.

To be fair, I haven't hit this limit myself, because I'm not serving hundreds of millions of users. But connections leak in ordinary apps too, through idle ones you forgot to close or one connection per request, so a pooler is worth having long before you're at OpenAI's size.

OpenAI serves most reads from a cache. The danger is a sudden drop in the hit rate, because every miss turns into a database query at the same moment.

Their fix is a lock per cache key. When many requests miss on the same key, one takes the lock, reads from Postgres and refills the cache. The rest wait for the cache instead of all going to the database. I think this is genius, because the data only needs to come from the database once. If you're caching LLM answers rather than rows, I wrote about semantic caching separately.

The primary streams its write-ahead log (WAL), the record of every change, to each replica. With nearly 50 replicas that's a lot of network and CPU on one machine, and replica lag gets worse as you add more.

The fix is cascading replication: the primary sends WAL to a few intermediate replicas, and they pass it on to the rest. OpenAI says this could take them past a hundred replicas. In the video I talked about it as if it's already running, but it isn't. They're still testing it with the Azure team, because it makes failover harder.

OpenAI rate-limits at the application, the connection pooler, the proxy and the query layer. They also changed their ORM so it can rate-limit or fully block a specific query pattern, which lets them cut off a runaway expensive query fast.

They also warn against short retry intervals, because those cause retry storms. The post doesn't give numbers, so here's what I'd do: wait 5 or 10 seconds before the first retry, back off to 30 or 60 seconds if it keeps failing, add some randomness so clients don't retry in sync, and after a few failures tell the user something's wrong and alert yourself.

Some schema changes, like changing a column's type, make Postgres rewrite the whole table. OpenAI only allows lightweight changes, such as adding or removing certain columns. Creating and dropping indexes concurrently is allowed. Every schema change has a strict 5-second timeout. When they backfill a column, the rate limit is so tight that it sometimes takes over a week.

A week sounds absurd when you can change a schema in seconds on a small app. But it never causes downtime, and at their scale I think that's the right trade.

Here's how to see it for yourself. Start with a table of a million rows:

CREATE TABLE users (
  id    bigint PRIMARY KEY,
  email text NOT NULL,
  age   integer
);

INSERT INTO users (id, email, age)
SELECT g, 'user' || g || '@example.com', 20 + g % 50
FROM generate_series(1, 1000000) AS g;

Then compare the table's file number before and after each change. A new number means Postgres wrote a whole new copy of the table:

\timing on
SELECT pg_relation_filenode('users');

-- Metadata only: no rewrite
ALTER TABLE users ADD COLUMN plan text DEFAULT 'free';
SELECT pg_relation_filenode('users');

-- Changing the type rewrites every row
ALTER TABLE users ALTER COLUMN age TYPE bigint;
SELECT pg_relation_filenode('users');

Run it with psql -f. Adding the column took 6 ms and the file number stayed at 16384. Changing age to bigint took 3.3 seconds and the number changed to 16394. That was on a million small rows on my machine. On a big production table it can run for minutes, and the table is locked the whole time.

OpenAI doesn't say which timeout setting they use. I'd set two at the top of every migration:

SET lock_timeout = '2s';
SET statement_timeout = '5s';

ALTER TABLE users ADD COLUMN email_domain text;

lock_timeout matters even for instant changes. An ALTER TABLE needs an exclusive lock, so it waits behind any open transaction on the table, and every query that arrives after it queues behind the ALTER. In my test, one transaction sat open on users and an ALTER TABLE without a timeout went into the queue. A one-row lookup by primary key then took 6.5 seconds. With the timeouts above, the migration gave up after 2 seconds with canceling statement due to lock timeout, and the same lookup took 1.7 seconds. You run the migration again later.

To fill the new column, update a few thousand rows at a time and between batches:

#!/usr/bin/env bash
set -euo pipefail

max_id=$(psql "$DATABASE_URL" -qtAc "SELECT max(id) FROM users")

for ((start = 1; start <= max_id; start += 5000)); do
  psql "$DATABASE_URL" -qc "
    UPDATE users
    SET email_domain = split_part(email, '@', 2)
    WHERE id >= $start AND id < $start + 5000
      AND email_domain IS NULL"
  sleep 0.2
done

echo "done up to id $max_id"

Run it with your connection string in DATABASE_URL:

DATABASE_URL="postgresql://user:password@localhost:5432/postgres" ./backfill.sh

It printed done up to id 1000000 after 72 seconds, 40 of them spent in the s. A single UPDATE did the same million rows in 9 seconds, but it locks every row until it commits and writes all its WAL in one burst that every replica then has to replay. Raise the sleep if your database is busy. OpenAI's backfills sometimes take over a week.

OpenAI reports low double-digit millisecond p99 latency on the client side and five nines of availability. In the past 12 months they had one SEV-0 Postgres incident, during the ChatGPT ImageGen launch, when write traffic jumped more than 10x as over 100 million new users signed up in a week. They're still moving the remaining write-heavy workloads off Postgres, and they haven't ruled out sharding it later.

If you're a backend developer, this post is gold. Most of us reach for a new database the moment an app gets a few thousand users, and OpenAI is running ChatGPT on one Postgres primary. Most of their fixes work on a small app too. Reading the SQL your ORM writes and putting a timeout on every migration cost you an afternoon, and neither needs a new database. Go read their post too, because they explain all of it better than I did in the video.

Originally published on hassandev.me.

── more in #ai-infrastructure 4 stories · sorted by recency
── more on @openai 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/how-openai-scales-po…] indexed:0 read:10min 2026-11-18 · —