{"slug": "how-chatgpt-serves-900-million-users-at-a-time", "title": "How ChatGPT Serves 900 Million Users at a Time", "summary": "OpenAI's ChatGPT now serves 900 million weekly active users, processing about 29,000 messages per second, according to a developer's analysis of the platform's architecture. The system relies on a global router, load balancers, and a PostgreSQL database with a single writer and 50 read replicas to handle the massive read-heavy workload. The developer highlights caching and careful scaling as key techniques that keep latency low despite the enormous traffic.", "body_md": "*Hello, I'm Maneshwar. I'm building git-lrc, a Micro AI code reviewer that runs on every commit. It is free and source-available on Github. Star git-lrc to help devs discover the project. Do give it a try and share your feedback.*\n\nRight now, as you read this sentence, roughly 900 million other people are also poking at ChatGPT every week, and a big chunk of them are typing at the exact same second you are.\n\nOpenAI reported crossing 900 million weekly active users back in February 2026, with something like 2.5 billion messages flying in per day.\n\nThat is around 29,000 messages *every single second*.\n\nAnd yet you hit enter and get an answer in a couple of seconds.\n\nNo spinning wheel of doom. No \"please try again later.\"\n\nI was trying to understand how that actually works, and honestly the answer is less \"alien technology\" and more \"a lot of boring ideas stacked really carefully.\"\n\nLet me walk you through the journey your little message takes, because it is a genuinely lovely piece of engineering and there are a couple of tricks in here you can steal for your own apps today.\n\nGrab a coffee. We are going server hopping.\n\nYour message does not go straight to \"ChatGPT.\"\n\nThat would be like everyone in the world trying to walk through one door.\n\nInstead it lands on a **global router**.\n\nThink of it as a very opinionated bouncer who looks at your request and decides which regional data center should handle you.\n\nIt weighs a few things: where you physically are, how much spare compute each region has, and what kind of hardware your request needs.\n\nSo a request from Bengaluru and a request from New York will very likely get sent to completely different regions. (India is actually OpenAI's second largest market now, with around 100 million weekly users, so that Bengaluru request is far from alone.)\n\nThe whole point is to keep you close to the compute and away from the traffic jams.\n\nOnce you land in a region, there is not one big heroic application server catching everything.\n\nThere are *hundreds* of them, and a **load balancer** sprays traffic across them so no single box gets flattened.\n\nEach of those app servers does the unglamorous but critical prep work before your message ever sniffs an AI model:\n\nThe beautiful part of this layer is that scaling it is almost embarrassingly easy.\n\nTraffic spikes? Add more servers.\n\nIt is the closest thing our industry has to a cheat code.\n\nBut here is the plot twist that makes this whole story interesting: **adding servers is the easy bit. Feeding them data is where it gets spicy.**\n\nOkay, brace yourself, because this is the part that made me laugh out loud.\n\nThe database behind one of the most futuristic products on earth is... plain old **PostgreSQL**. No exotic distributed NewSQL wizardry. No blockchain (thank goodness).\n\nJust Postgres, a database originally cooked up by researchers at UC Berkeley, pushed to an absolutely heroic degree.\n\nThe setup, straight from [OpenAI's own engineering post](https://openai.com/index/scaling-postgresql/), is delightfully simple on paper:\n\nThat is it. One writer. Fifty readers. Serving millions of queries per second at low double digit millisecond latency and five nines of availability.\n\nThe reason this works is that ChatGPT's workload is wildly read heavy.\n\nYou send the occasional message, but the system is constantly *reading*: your history, your settings, your permissions, model configs. Reads you can fan out across replicas forever.\n\nWrites are the hard part, which is why they guard that single primary like it owes them money.\n\nHow do they keep one lonely primary alive under all this? Two moves, working together.\n\nHere is the single most important idea in this whole post, and it costs you nothing to adopt: **most of those database reads should never touch the database at all.**\n\nIf a thousand requests all need the same piece of info, you do not want a thousand trips to Postgres.\n\nYou want *one* trip.\n\nOne request fetches the data, stashes it in an in memory cache, and the other 999 read it straight from that cache.\n\nFast, cheap, and the database barely notices.\n\nBut caches have a nasty failure mode, and this is where OpenAI does something clever.\n\nImagine a popular cache entry suddenly expires or the cache layer hiccups. Now a thousand requests all \"miss\" at the same instant and stampede toward Postgres simultaneously.\n\nThis is a real, named disaster called a [cache stampede](https://en.wikipedia.org/wiki/Cache_stampede) (or the \"thundering herd\"), and it has genuinely taken services down.\n\nOpenAI's fix is a **cache lock**.\n\nWhen a bunch of requests miss the same key at once, only *one* of them gets the lock and is allowed to go ask Postgres.\n\nEveryone else just... waits for that one to come back and refill the cache.\n\nThe herd gets politely told to form an orderly queue.\n\nIf you take one thing home from this post, make it this: caching is not just a speed optimization, it is *load bearing*.\n\nIt is what lets a single Postgres primary sleep at night.\n\nYou have been authenticated, your history is loaded, your data is cached and ready.\n\nOnly now do you reach the actual model.\n\nAnd there is another scheduler waiting.\n\nThe **inference scheduler** decides which GPU cluster should run your request.\n\nIt is basically playing a giant game of Tetris, looking at:\n\nThat last one is sneaky smart.\n\nIf some of your context is already on a machine, sending you back there saves a ton of recomputation.\n\nInside a cluster, here is the trick that makes the economics work at all: **batching**.\n\nYour request does not get its own private forward pass through the model.\n\nIt gets bundled together with a pile of other people's requests, and one pass through the model serves the whole batch at once.\n\nYou are sharing a ride with strangers and none of you can tell.\n\nFor the really big models, one GPU is not enough to hold the whole thing, so the model is split across several GPUs.\n\nEach one solves its slice of the problem and the results get stitched back together. It is teamwork, but for silicon.\n\nHere is a detail I love because it is half engineering, half psychology.\n\nThe model does not compute your entire answer, wrap it in a bow, and then hand it over.\n\nAs soon as it generates the first token, ChatGPT streams it straight to your screen.\n\nThat is why you see the answer type itself out word by word instead of staring at a blank box for eight seconds.\n\nFunctionally it means you start reading before the model has even finished thinking.\n\nPerceptually it makes the whole thing feel alive and fast.\n\nNo buffering, no waiting, just a steady drip of tokens.\n\nA small token of appreciation for your patience, if you will.\n\nLast piece.\n\nRemember that fragile single primary? OpenAI protects it with **rate limits at four separate layers**: the application, the connection pooler, the proxy, and the query level itself.\n\nWhy four? Because the scariest thing for a database is not steady heavy traffic, it is a *sudden* spike.\n\nA burst of expensive queries, or a retry storm where failing requests all retry at once and pile on even more load, can spiral into a full outage.\n\n(OpenAI mentions their only serious Postgres incident in a year happened during the viral ImageGen launch, when write traffic jumped more than 10x as over 100 million new users showed up in a single week. Even the boring database has war stories.)\n\nRate limiting at every layer means a bad spike gets absorbed early instead of cascading all the way down to the one machine that cannot take it.\n\nHere is the full trip your message takes, start to finish:\n\nThe thing that stuck with me is how *unsexy* the winning moves are.\n\nThere is no secret sauce here that you cannot use in your own weekend project:\n\nIf you want to go deeper, OpenAI's [own writeup](https://openai.com/index/scaling-postgresql/) is genuinely readable, and [ByteByteGo's breakdown](https://blog.bytebytego.com/p/how-openai-scaled-to-800-million) is a great visual companion.\n\nWhat is the most \"boring tech, wild scale\" story you have run into? Drop it in the comments, I collect these.\n\nAI agents write code fast. They also silently remove logic, change behavior, and introduce bugs — without telling you. You often find out in production.\n\ngit-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.\n\nAny feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.\n\n⭐ Star it on GitHub:\n\n| [🇩🇰 Dansk](https://github.com/HexmosTech/git-lrc/readme/README.da.md) | [🇪🇸 Español](https://github.com/HexmosTech/git-lrc/readme/README.es.md) | [🇮🇷 Farsi](https://github.com/HexmosTech/git-lrc/readme/README.fa.md) | [🇫🇮 Suomi](https://github.com/HexmosTech/git-lrc/readme/README.fi.md) | [🇯🇵 日本語](https://github.com/HexmosTech/git-lrc/readme/README.ja.md) | [🇳🇴 Norsk](https://github.com/HexmosTech/git-lrc/readme/README.nn.md) | [🇵🇹 Português](https://github.com/HexmosTech/git-lrc/readme/README.pt.md) | [🇷🇺 Русский](https://github.com/HexmosTech/git-lrc/readme/README.ru.md) | [🇦🇱 Shqip](https://github.com/HexmosTech/git-lrc/readme/README.sq.md) | [🇨🇳 中文](https://github.com/HexmosTech/git-lrc/readme/README.zh.md) | [🇮🇳 हिन्दी](https://github.com/HexmosTech/git-lrc/readme/README.hi.md) |\n\nGenAI today is a **race car without brakes**. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents *silently break things*: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.\n\n** git-lrc is your braking system.** It hooks into\n\n`git commit`\n\nand runs an AI review on every diff In short, git-lrc helps **Prevent Outages, Breaches, and Technical Debt Before They Happen**\n\n**At a glance:** [10 risk categories](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · [100+ failure patterns tracked](https://github.com/HexmosTech/git-lrc#what-git-lrc-checks-for) · every commit…", "url": "https://wpnews.pro/news/how-chatgpt-serves-900-million-users-at-a-time", "canonical_source": "https://dev.to/lovestaco/how-chatgpt-serves-900-million-users-at-a-time-64h", "published_at": "2026-08-21 17:04:40+00:00", "updated_at": "2026-08-21 17:15:06.120796+00:00", "lang": "en", "topics": ["ai-infrastructure", "large-language-models", "developer-tools"], "entities": ["OpenAI", "ChatGPT", "PostgreSQL", "UC Berkeley", "Maneshwar"], "alternates": {"html": "https://wpnews.pro/news/how-chatgpt-serves-900-million-users-at-a-time", "markdown": "https://wpnews.pro/news/how-chatgpt-serves-900-million-users-at-a-time.md", "text": "https://wpnews.pro/news/how-chatgpt-serves-900-million-users-at-a-time.txt", "jsonld": "https://wpnews.pro/news/how-chatgpt-serves-900-million-users-at-a-time.jsonld"}}