The Postgres Creator Says LLMs Score 0% on Real Databases

Turing Award winner and Postgres creator Mike Stonebraker revealed that large language models score 0% accuracy on real-world data warehouse queries, despite achieving 80-85% on popular benchmarks like Spider and Bird. Stonebraker tested models on four production databases and found zero correct queries, rising to only 10% with RAG and 35% when given exact schema details. He attributes the failure to enterprise data being absent from training sets, higher query complexity, messy real-world schemas, and idiosyncratic domain-specific concepts.

Mike Stonebraker has been right before. He built Ingres in 1972, then Postgres in the 1980s, then predicted the death of general-purpose databases in 2005. Every time, the industry caught up years later. Now he is saying something that should make every company betting on "chat with your data" very nervous. In a recent interview on the Data Renegades podcast, the Turing Award winner revealed that LLMs score 0% accuracy on real-world data warehouse queries. Not 80%, which is what the popular benchmarks report. Zero. The text-to-SQL leaderboard tells a reassuring story. On popular benchmarks like Spider and Bird, the best LLM systems hit 80-85% accuracy. Impressive. Almost production-ready. Stonebraker tested the same models on four real production data warehouses. Not synthetic data. Not academic datasets. Actual enterprise systems with actual workloads from actual users. The result: 0% accuracy . Add RAG and every trick in the book: 10% . Hand the model the exact FROM clause and all JOIN terms on a silver platter: 35% . A knowledgeable SQL programmer with access to the schema? 90%+. This is not a small gap. This is the difference between "almost there" and "does not work at all." Stonebraker identified four reasons the benchmarks are meaningless for production use: 1. Enterprise data is not in the training set. LLMs are trained on "the pile," the massive web corpus. Your data warehouse is not in the pile. And if an LLM has not seen data a couple times before, it has almost no chance of reasoning about it correctly. 2. Real queries are 5x more complex. Spider and Bird benchmarks use queries that are 10-20 lines of SQL. Real data warehouse queries run 100+ lines. The complexity scales non-linearly. 3. Real schemas are a mess. Academic benchmarks have clean, mnemonic table and column names like employees.salary . Real data warehouses have materialized views everywhere creating redundancy , column names like ZUPPERS BLAH , and idiosyncratic naming conventions built up over decades. 4. Idiosyncratic data. Stonebraker gives the example of "J-term" at MIT, a one-month January term. Not unique to MIT, but not common enough to appear in training data. Every enterprise has dozens of such domain-specific concepts that no LLM has ever seen. Stonebraker published these benchmarks as BEAVER, an anonymized version of the four real data warehouses, available on arXiv https://arxiv.org/abs/2409.02038 . His message to AI researchers: "If you think you're really good at text-to-SQL, try a real benchmark, not a fake one." Stonebraker has seen tech hype cycles before. He built Ingres at Berkeley in 1972 and commercialized it in 1980. The competition was Larry Ellison's Oracle. His assessment of how Oracle competed is blunt. "Larry Ellison is a fabulous salesman. He made present tense and future tense indistinguishable. He basically lied to customers." The example he gives is telling. Ingres implemented referential integrity, the database constraint that ensures data consistency if you fire the last employee in a department, should the department still exist? . Oracle wrote two manual pages defining referential integrity, then added at the bottom: "Not yet implemented." Customers bought Oracle anyway. The feature was on the roadmap. The documentation existed. The code did not. Sound familiar? It is the same playbook playing out today with AI companies shipping demos of capabilities that do not work in production. In 2004, Stonebraker published a paper arguing that general-purpose databases were doomed. "One Size Fits All: An Idea Whose Time Has Come and Gone" Brown University PDF https://cs.brown.edu/people/ugur/fits all.pdf showed that specialized engines outperformed general-purpose systems by an order of magnitude in specific workloads. Column stores like Vertica and ClickHouse destroy row stores for analytics. Stream processors like StreamBase which Stonebraker also built bear no resemblance to relational databases. Pinecone is faster than user-defined types on Postgres for vector search. His take on Postgres today is surprisingly honest: "At the low end, it's absolutely the right one-size-fits-all. At the high end, that's just not true." Postgres has no column store, no multi-node support, and is not competitive for sizable data warehouses. For getting started, it is the right choice. For petabyte-scale analytics, it is table stakes that it does not have. At 83, Stonebraker has no incentive to be cautious. When asked what he would major in if starting today, his answer was blunt: "Computer science may well not be a growth industry going forward. I'm not sure I would recommend 18-year-olds to major in computer science. Health care and the building trades are safe bets. Everything else looks much riskier." This from the man who built the database that runs much of the internet. His reasoning is consistent with his entire career. He has always bet against incumbents and conventional wisdom. The conventional wisdom in 2026 is "learn to code, AI will just make you faster." Stonebraker's counter: if AI makes every coder 10x faster, you need 10x fewer coders. The math is not complicated. Stonebraker's current project is DBOS Database-Oriented Operating System , born from a collaboration with Matei Zaharia, the creator of Apache Spark and co-founder of Databricks. The insight: Databricks was scheduling a million Spark jobs at any given time, and no existing OS scheduler could handle that scale. So they put all the scheduling data in Postgres and let a database application do the scheduling. Then it clicked. Most of what an operating system does is manage data at scale. Why not replace the upper half of Linux with a database? The academic project showed that a file system built on top of a DBMS is faster than the Linux file system. The scheduling engine is competitive. Everything fails over automatically. High availability comes for free. When Stonebraker mentioned this to OS folks, "they got very, very threatened. The database guys are trying to take over their turf." Same reaction from programming language folks when he suggested the runtime for a programming environment should be a database. DBOS is now a commercial product dbos.dev https://www.dbos.dev/ , offering durable workflows in TypeScript, Java, Go, and Python. Two-thirds of their customers are building agentic AI. The key insight: most agentic AI today is read-only generate a prediction . But it is moving to read-write, and that is a distributed database problem. You want atomicity, consistency, and transactions. Moving $100 between accounts using two AI agents requires both to commit or both to roll back. That is what databases were built for. Three takeaways from Stonebraker's career and current work: 1. "Chat with your database" is not production-ready. The benchmarks are gamed. Real enterprise data is messy, complex, and full of domain-specific knowledge that LLMs have never seen. If you are building a product that depends on text-to-SQL working reliably, test it on your actual data warehouse, not Spider. 2. Specialized always wins at scale. Postgres is the right choice for getting started. But if you are running a petabyte data warehouse, you need a column store. If you are doing vector search, you need a vector database. If you are processing streams, you need a stream processor. The one-size-fits-all era is over at the high end. 3. Agentic AI needs database fundamentals. When AI agents start writing to databases, you need transactions, consistency, and durability. That is not an AI problem. It is a database problem. The engineers who understand both worlds, AI and database internals, will be the ones building the systems that actually work. Stonebraker's career is a masterclass in betting against the herd. He was right about specialized databases. He was right about MapReduce being inefficient Google eventually abandoned it . He was right about eventual consistency being wrong for most use cases Google abandoned that too with Spanner . His track record on text-to-SQL should make you pause before trusting the benchmark numbers. Sources: Data Renegades Podcast - Mike Stonebraker Interview, BEAVER Benchmark arXiv , Recce Blog: Benchmarks Lie, One Size Fits All paper Brown University , DBOS, DBOS Wikipedia