{"slug": "how-openai-scales-postgresql-to-800-million-users", "title": "How OpenAI scales PostgreSQL to 800 million users", "summary": "OpenAI scales a single PostgreSQL primary, backed by nearly 50 cross-region read replicas, to serve ChatGPT's 800 million users at millions of queries per second, according to the company's January 2026 engineering post. Rather than shard the existing deployment, OpenAI has frozen new tables in Postgres and routes new features to sharded systems like Azure Cosmos DB, while moving write-heavy workloads off the primary. The company attributes most outages to a retry-driven overload loop and counters it with replica reads, lazy writes, rate-limited backfills, transaction timeouts and a hot standby.", "body_md": "OpenAI scales PostgreSQL for ChatGPT with one primary server that takes every write and nearly 50 read replicas spread across regions, and that handles millions of queries per second for 800 million users. Most of the work goes into keeping load off the primary, and into limits at every layer so one spike can't snowball into an outage.\n\nBelow is each problem they hit and how they fixed it, checked against [OpenAI's own engineering post](https://openai.com/index/scaling-postgresql/) from January 2026, plus the parts I'd copy in a normal app, with SQL I tested on Postgres 18. The video above covers the same ground.\n\nThis came as a shock to me. I had no idea a single Postgres write instance could go that far. The primary is an Azure Database for PostgreSQL flexible server, and OpenAI says its Postgres load grew more than 10x in the past year.\n\nThey haven't sharded it, because sharding the existing workloads would mean changing hundreds of application endpoints and could take months or years. Instead they've drawn a line: no new tables go into this Postgres deployment, and new features default to sharded systems like Azure Cosmos DB.\n\nWrites are the weak spot. Postgres uses multiversion concurrency control (MVCC), so updating even one field copies the whole row into a new version. Under heavy writes that means write amplification, dead rows that queries have to skip past, table and index bloat, and autovacuum work. Reads scale out across replicas. Writes all land on one machine.\n\nOpenAI says its Postgres outages usually follow the same loop. Something upstream causes a sudden jump in load. The cache layer fails and every request misses, or a new feature launch floods the database with writes. A burst of expensive multi-table joins eating the CPU can do it too. Queries slow down and start timing out. Clients retry, the retries add even more load, and it keeps getting worse until the whole service is down.\n\nEvery fix below breaks that loop somewhere.\n\nReads go to replicas wherever possible. The only reads left on the primary are the ones inside write transactions, and those get tuned so they stay fast.\n\nFor writes, OpenAI moved the shardable, write-heavy workloads to Cosmos DB. The harder-to-shard ones are still being moved. Then they went through their own code. They fixed bugs that wrote the same data twice and added lazy writes to spread spikes out. Backfills got strict rate limits, so filling in a column can't hammer the primary (more on that below).\n\nYou can hunt for duplicate writes in a small app the same way. Turn on the `pg_stat_statements` extension, sort by how often each query runs, and look for writes that run more often than they should.\n\nOpenAI found a query that joined 12 tables, and spikes in it caused some of their worst incidents. Their advice is to avoid multi-table joins where you can, and to break up the ones you need and do the join in application code instead.\n\nThey also say many of the bad queries came from ORMs. Prisma, Drizzle, SQLAlchemy and Django's ORM all make it easy to write code that turns into slow SQL without you noticing, so read the SQL your ORM generates for your hottest endpoints.\n\nThen set a timeout for transactions that sit open doing nothing. OpenAI uses `idle_in_transaction_session_timeout`, because those sessions stop autovacuum from cleaning up dead rows:\n\n```\nALTER DATABASE postgres SET idle_in_transaction_session_timeout = '60s';\n```\n\nSwap `postgres` for your database name. New connections pick it up. When I tested it with a 3-second value, a session that ran `BEGIN` and then sat idle got cut off with `FATAL: terminating connection due to idle-in-transaction timeout`.\n\nA single writer is a single point of failure. OpenAI runs the primary in high-availability mode with a hot standby, a replica kept in sync that can be promoted quickly if the primary dies or needs maintenance. In the video I called this zero downtime, but the post only promises to keep downtime short.\n\nThe bigger fix is that most critical requests only read data, and those reads now go to replicas. If the primary goes down, writes fail but ChatGPT keeps answering reads, so it's no longer their worst severity of incident. They also run more than one replica in every region, so losing one replica doesn't take a region down.\n\nSometimes one workload, like a new feature with inefficient queries, eats most of the CPU and slows everything else on the same instance. OpenAI splits requests into high-priority and low-priority tiers and routes each tier to its own instances. A heavy low-priority job can then max out its own servers without touching high-priority traffic. They do the same between products, so one product's traffic can't hurt another.\n\nEach Azure PostgreSQL instance allows at most 5,000 connections, and OpenAI has had incidents where a connection storm used them all up. (Plain Postgres has no fixed number. `max_connections` is a setting, 100 by default.)\n\nTheir fix is PgBouncer in statement or transaction pooling mode, so many clients share a few real connections. In their benchmarks the average time to get a connection dropped from 50 ms to 5 ms. PgBouncer runs in the same region as the clients and the replica it sits in front of, and every read replica has its own Kubernetes deployment running several PgBouncer pods behind one Kubernetes Service.\n\nTo be fair, I haven't hit this limit myself, because I'm not serving hundreds of millions of users. But connections leak in ordinary apps too, through idle ones you forgot to close or one connection per request, so a pooler is worth having long before you're at OpenAI's size.\n\nOpenAI serves most reads from a cache. The danger is a sudden drop in the hit rate, because every miss turns into a database query at the same moment.\n\nTheir fix is a lock per cache key. When many requests miss on the same key, one takes the lock, reads from Postgres and refills the cache. The rest wait for the cache instead of all going to the database. I think this is genius, because the data only needs to come from the database once. If you're caching LLM answers rather than rows, I wrote about [semantic caching](https://www.hassandev.me/blog/youtube/semantic-caching-stop-paying-for-the-same-answer-3gU_BA_19f0) separately.\n\nThe primary streams its write-ahead log (WAL), the record of every change, to each replica. With nearly 50 replicas that's a lot of network and CPU on one machine, and replica lag gets worse as you add more.\n\nThe fix is cascading replication: the primary sends WAL to a few intermediate replicas, and they pass it on to the rest. OpenAI says this could take them past a hundred replicas. In the video I talked about it as if it's already running, but it isn't. They're still testing it with the Azure team, because it makes failover harder.\n\nOpenAI rate-limits at the application, the connection pooler, the proxy and the query layer. They also changed their ORM so it can rate-limit or fully block a specific query pattern, which lets them cut off a runaway expensive query fast.\n\nThey also warn against short retry intervals, because those cause retry storms. The post doesn't give numbers, so here's what I'd do: wait 5 or 10 seconds before the first retry, back off to 30 or 60 seconds if it keeps failing, add some randomness so clients don't retry in sync, and after a few failures tell the user something's wrong and alert yourself.\n\nSome schema changes, like changing a column's type, make Postgres rewrite the whole table. OpenAI only allows lightweight changes, such as adding or removing certain columns. Creating and dropping indexes concurrently is allowed. Every schema change has a strict 5-second timeout. When they backfill a column, the rate limit is so tight that it sometimes takes over a week.\n\nA week sounds absurd when you can change a schema in seconds on a small app. But it never causes downtime, and at their scale I think that's the right trade.\n\nHere's how to see it for yourself. Start with a table of a million rows:\n\n```\nCREATE TABLE users (\n  id    bigint PRIMARY KEY,\n  email text NOT NULL,\n  age   integer\n);\n\nINSERT INTO users (id, email, age)\nSELECT g, 'user' || g || '@example.com', 20 + g % 50\nFROM generate_series(1, 1000000) AS g;\n```\n\nThen compare the table's file number before and after each change. A new number means Postgres wrote a whole new copy of the table:\n\n```\n\\timing on\nSELECT pg_relation_filenode('users');\n\n-- Metadata only: no rewrite\nALTER TABLE users ADD COLUMN plan text DEFAULT 'free';\nSELECT pg_relation_filenode('users');\n\n-- Changing the type rewrites every row\nALTER TABLE users ALTER COLUMN age TYPE bigint;\nSELECT pg_relation_filenode('users');\n```\n\nRun it with `psql -f`. Adding the column took 6 ms and the file number stayed at 16384. Changing `age` to `bigint` took 3.3 seconds and the number changed to 16394. That was on a million small rows on my machine. On a big production table it can run for minutes, and the table is locked the whole time.\n\nOpenAI doesn't say which timeout setting they use. I'd set two at the top of every migration:\n\n```\nSET lock_timeout = '2s';\nSET statement_timeout = '5s';\n\nALTER TABLE users ADD COLUMN email_domain text;\n```\n\n`lock_timeout` matters even for instant changes. An `ALTER TABLE` needs an exclusive lock, so it waits behind any open transaction on the table, and every query that arrives after it queues behind the `ALTER`. In my test, one transaction sat open on `users` and an `ALTER TABLE` without a timeout went into the queue. A one-row lookup by primary key then took 6.5 seconds. With the timeouts above, the migration gave up after 2 seconds with `canceling statement due to lock timeout`, and the same lookup took 1.7 seconds. You run the migration again later.\n\nTo fill the new column, update a few thousand rows at a time and pause between batches:\n\n``` bash\n#!/usr/bin/env bash\n# Fill users.email_domain 5,000 rows at a time, with a pause between batches.\nset -euo pipefail\n\nmax_id=$(psql \"$DATABASE_URL\" -qtAc \"SELECT max(id) FROM users\")\n\nfor ((start = 1; start <= max_id; start += 5000)); do\n  psql \"$DATABASE_URL\" -qc \"\n    UPDATE users\n    SET email_domain = split_part(email, '@', 2)\n    WHERE id >= $start AND id < $start + 5000\n      AND email_domain IS NULL\"\n  sleep 0.2\ndone\n\necho \"done up to id $max_id\"\n```\n\nRun it with your connection string in `DATABASE_URL`:\n\n```\nDATABASE_URL=\"postgresql://user:password@localhost:5432/postgres\" ./backfill.sh\n```\n\nIt printed `done up to id 1000000` after 72 seconds, 40 of them spent in the pauses. A single `UPDATE` did the same million rows in 9 seconds, but it locks every row until it commits and writes all its WAL in one burst that every replica then has to replay. Raise the `sleep` if your database is busy. OpenAI's backfills sometimes take over a week.\n\nOpenAI reports low double-digit millisecond p99 latency on the client side and five nines of availability. In the past 12 months they had one SEV-0 Postgres incident, during the ChatGPT ImageGen launch, when write traffic jumped more than 10x as over 100 million new users signed up in a week. They're still moving the remaining write-heavy workloads off Postgres, and they haven't ruled out sharding it later.\n\nIf you're a backend developer, this post is gold. Most of us reach for a new database the moment an app gets a few thousand users, and OpenAI is running ChatGPT on one Postgres primary. Most of their fixes work on a small app too. Reading the SQL your ORM writes and putting a timeout on every migration cost you an afternoon, and neither needs a new database. Go read [their post](https://openai.com/index/scaling-postgresql/) too, because they explain all of it better than I did in the video.\n\n*Originally published on [hassandev.me](https://www.hassandev.me/blog/youtube/800-million-users-1-database-r_7KmsmepBc).*", "url": "https://wpnews.pro/news/how-openai-scales-postgresql-to-800-million-users", "canonical_source": "https://dev.to/hassan_aziz_88b11714fef8b/how-openai-scales-postgresql-to-800-million-users-1cl1", "published_at": "2026-11-18 09:38:58+00:00", "updated_at": "2026-11-18 09:43:48.124856+00:00", "lang": "en", "topics": ["ai-infrastructure", "mlops", "ai-products"], "entities": ["OpenAI", "PostgreSQL", "ChatGPT", "Azure Database for PostgreSQL", "Azure Cosmos DB", "Prisma", "Drizzle", "SQLAlchemy"], "also_reported_by": [], "takeaway": "OpenAI keeps writes on one Postgres primary and scales reads across ~50 replicas, breaking overload loops with idle_in_transaction_session_timeout, lazy writes, rate-limited backfills and moving write-heavy workloads to Cosmos DB.", "recommendation": {"advice": "Set idle_in_transaction_session_timeout (OpenAI uses 60s) and rate-limit backfills so a load spike can't trigger the retry loop that takes the primary down.", "playbook": "production-ops", "playbook_url": "https://wpnews.pro/playbooks/production-ops"}, "alternates": {"html": "https://wpnews.pro/news/how-openai-scales-postgresql-to-800-million-users", "markdown": "https://wpnews.pro/news/how-openai-scales-postgresql-to-800-million-users.md", "text": "https://wpnews.pro/news/how-openai-scales-postgresql-to-800-million-users.txt", "jsonld": "https://wpnews.pro/news/how-openai-scales-postgresql-to-800-million-users.jsonld"}}