# How ChatGPT Serves 900 Million Users at a Time

> Source: <https://dev.to/lovestaco/how-chatgpt-serves-900-million-users-at-a-time-64h>
> Published: 2026-08-21 17:04:40+00:00

*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*

Right now, as you read this sentence, roughly 900 million other people are also poking at ChatGPT every week, and a big chunk of them are typing at the exact same second you are.

OpenAI reported crossing 900 million weekly active users back in February 2026, with something like 2.5 billion messages flying in per day.

That is around 29,000 messages *every single second*.

And yet you hit enter and get an answer in a couple of seconds.

No spinning wheel of doom. No "please try again later."

I was trying to understand how that actually works, and honestly the answer is less "alien technology" and more "a lot of boring ideas stacked really carefully."

Let me walk you through the journey your little message takes, because it is a genuinely lovely piece of engineering and there are a couple of tricks in here you can steal for your own apps today.

Grab a coffee. We are going server hopping.

Your message does not go straight to "ChatGPT."

That would be like everyone in the world trying to walk through one door.

Instead it lands on a **global router**.

Think of it as a very opinionated bouncer who looks at your request and decides which regional data center should handle you.

It weighs a few things: where you physically are, how much spare compute each region has, and what kind of hardware your request needs.

So a request from Bengaluru and a request from New York will very likely get sent to completely different regions. (India is actually OpenAI's second largest market now, with around 100 million weekly users, so that Bengaluru request is far from alone.)

The whole point is to keep you close to the compute and away from the traffic jams.

Once you land in a region, there is not one big heroic application server catching everything.

There are *hundreds* of them, and a **load balancer** sprays traffic across them so no single box gets flattened.

Each of those app servers does the unglamorous but critical prep work before your message ever sniffs an AI model:

The beautiful part of this layer is that scaling it is almost embarrassingly easy.

Traffic spikes? Add more servers.

It is the closest thing our industry has to a cheat code.

But here is the plot twist that makes this whole story interesting: **adding servers is the easy bit. Feeding them data is where it gets spicy.**

Okay, brace yourself, because this is the part that made me laugh out loud.

The database behind one of the most futuristic products on earth is... plain old **PostgreSQL**. No exotic distributed NewSQL wizardry. No blockchain (thank goodness).

Just Postgres, a database originally cooked up by researchers at UC Berkeley, pushed to an absolutely heroic degree.

The setup, straight from [OpenAI's own engineering post](https://openai.com/index/scaling-postgresql/), is delightfully simple on paper:

That is it. One writer. Fifty readers. Serving millions of queries per second at low double digit millisecond latency and five nines of availability.

The reason this works is that ChatGPT's workload is wildly read heavy.

You send the occasional message, but the system is constantly *reading*: your history, your settings, your permissions, model configs. Reads you can fan out across replicas forever.

Writes are the hard part, which is why they guard that single primary like it owes them money.

How do they keep one lonely primary alive under all this? Two moves, working together.

Here is the single most important idea in this whole post, and it costs you nothing to adopt: **most of those database reads should never touch the database at all.**

If a thousand requests all need the same piece of info, you do not want a thousand trips to Postgres.

You want *one* trip.

One request fetches the data, stashes it in an in memory cache, and the other 999 read it straight from that cache.

Fast, cheap, and the database barely notices.

But caches have a nasty failure mode, and this is where OpenAI does something clever.

Imagine a popular cache entry suddenly expires or the cache layer hiccups. Now a thousand requests all "miss" at the same instant and stampede toward Postgres simultaneously.

This is a real, named disaster called a [cache stampede](https://en.wikipedia.org/wiki/Cache_stampede) (or the "thundering herd"), and it has genuinely taken services down.

OpenAI's fix is a **cache lock**.

When a bunch of requests miss the same key at once, only *one* of them gets the lock and is allowed to go ask Postgres.

Everyone else just... waits for that one to come back and refill the cache.

The herd gets politely told to form an orderly queue.

If you take one thing home from this post, make it this: caching is not just a speed optimization, it is *load bearing*.

It is what lets a single Postgres primary sleep at night.

You have been authenticated, your history is loaded, your data is cached and ready.

Only now do you reach the actual model.

And there is another scheduler waiting.

The **inference scheduler** decides which GPU cluster should run your request.

It is basically playing a giant game of Tetris, looking at:

That last one is sneaky smart.

If some of your context is already on a machine, sending you back there saves a ton of recomputation.

Inside a cluster, here is the trick that makes the economics work at all: **batching**.

Your request does not get its own private forward pass through the model.

It gets bundled together with a pile of other people's requests, and one pass through the model serves the whole batch at once.

You are sharing a ride with strangers and none of you can tell.

For the really big models, one GPU is not enough to hold the whole thing, so the model is split across several GPUs.

Each one solves its slice of the problem and the results get stitched back together. It is teamwork, but for silicon.

Here is a detail I love because it is half engineering, half psychology.

The model does not compute your entire answer, wrap it in a bow, and then hand it over.

As soon as it generates the first token, ChatGPT streams it straight to your screen.

That is why you see the answer type itself out word by word instead of staring at a blank box for eight seconds.

Functionally it means you start reading before the model has even finished thinking.

Perceptually it makes the whole thing feel alive and fast.

No buffering, no waiting, just a steady drip of tokens.

A small token of appreciation for your patience, if you will.

Last piece.

Remember that fragile single primary? OpenAI protects it with **rate limits at four separate layers**: the application, the connection pooler, the proxy, and the query level itself.

Why four? Because the scariest thing for a database is not steady heavy traffic, it is a *sudden* spike.

A burst of expensive queries, or a retry storm where failing requests all retry at once and pile on even more load, can spiral into a full outage.

(OpenAI mentions their only serious Postgres incident in a year happened during the viral ImageGen launch, when write traffic jumped more than 10x as over 100 million new users showed up in a single week. Even the boring database has war stories.)

Rate limiting at every layer means a bad spike gets absorbed early instead of cascading all the way down to the one machine that cannot take it.

Here is the full trip your message takes, start to finish:

The thing that stuck with me is how *unsexy* the winning moves are.

There is no secret sauce here that you cannot use in your own weekend project:

If you want to go deeper, OpenAI's [own writeup](https://openai.com/index/scaling-postgresql/) is genuinely readable, and [ByteByteGo's breakdown](https://blog.bytebytego.com/p/how-openai-scaled-to-800-million) is a great visual companion.

What is the most "boring tech, wild scale" story you have run into? Drop it in the comments, I collect these.

AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.

git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.

Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.

⭐ Star it on GitHub:

| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |

GenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.

** git-lrc is your braking system.** It hooks into

`git commit`

and runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**

**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…