{"slug": "database-architects-next-level-benchmarking-overfitting", "title": "Database Architects: Next-Level Benchmarking Overfitting", "summary": "Database Architects researchers analyzed GenDB-V, an AI-generated, formally verified database system whose paper claims 22× to 61× speedups over Umbra and DuckDB on TPC-H, and found the gains come from benchmark overfitting rather than genuine query-processing advances. The system precomputes per-query auxiliary structures, such as a supplier table ordered by s_acctbal DESC that makes Q2 near-constant time, and in its \"plain-array\" mode hard-codes knowledge of the TPC-H data generator's row layout to make Q18 joins trivial, falling back to a slow nested-loop path that never triggers on official data. The authors argue that letting AI agents optimize for a specific benchmark renders it useless, since the resulting tricks would not survive real workloads or data modifications.", "body_md": "In my [recent\nblog post](https://databasearchitects.blogspot.com/2026/10/the-benchmarking-crisis.html) I remarked that database benchmarking is in a crisis\nbecause AI agents start to overfit on individual benchmarks with\nquestionable techniques, which drives execution times towards zero. This\nhas happened with [TPC-D](https://www.tpc.org/tpcd/default5.asp) in the past, and\nbroke the benchmark. [TPC-H](https://www.tpc.org/tpch/default5.asp) tries to fight\nthat with rules, but most people who publish (unofficial) TPC-H results\nsimply ignore the rules. In combination with overly eager AI agents,\nthis leads to brutal overfitting.\n\nWe are seeing the next level of that with the [GenDB-V system](https://github.com/GenLang-V/GenDB-V). The\ncorresponding paper commands [“Stop Handcrafting Your Engine!\nGenDB-V: Towards AI-Generated and Formally Verified Database\nSystems”](https://arxiv.org/pdf/2611.02779), and reports speedups of 22× to 61× over Umbra and DuckDB,\nand all that in a fully verified system. These are indeed impressive\nclaims, and would suggest that traditional database engines are in\ntrouble.\n\nTaking a step back, though, it should be clear that a performance improvement of a factor of 22 over Umbra is infeasible, at least if we are talking about traditional query processing. At SF10, the 22 TPC-H queries read roughly 15GB of (compressed) base data, and on a 9950X3D Umbra runs all queries in a total of roughly 540ms. On that machine the memory touched for data divided by the memory bandwidth (54GB/s) alone is about 280ms; anything below that is infeasible. Better compression might bring that down a bit, but not by an order of magnitude.\n\nUnsurprisingly, the most dramatic gains of GenDB-V come from\nprecomputation. The agents have carefully handcrafted auxiliary data\nstructures for each query, which are computed at load time, and which\navoid most of the work. The physical design is heavily tuned to the\nqueries, too, for example, there is an extra structure that lists the\n`supplier` rows in `s_acctbal DESC` order, which\nmakes `Q2` trivial. Similar per-query data structures are\nbuilt for most queries. And we see that in the performance numbers:\nSeveral queries already run in near-constant time at SF100: Q2 in 0.7\nms, Q17 in about 15 ms and Q4 in about 17 ms on one thread. That is the\nTPC-D symptom, without stored aggregates.\n\nThere is also a “plain-array” mode, which, at first glance, is more\nfair, as it avoids these excessive preparations and also reports\nperformance improvements of just roughly a factor of 3, which seems\nachievable. But when looking closely, we see that here the agents did\nsomething even more insane: They [hard-coded](https://github.com/GenLang-V/GenDB-V/blob/a85741904b9fe6338827ea19d5b932fc389c07cc/results/tpch-sf100/gendb-v-plain-array-storage/queries/q18/code/lookup.rs#L31-L65)\ntheir knowledge about the data generator. They “know” that each customer\n`k` will be at row `k-1`. And they know that dbgen\nnumbers the `i`-th order `32·(i/8) + i%8`, so key\n`k` should be at row `8·(k/32) + k%32 − 1`. Which\nmakes the join in query `Q18` super cheap.\n\nHow does all this fit together with the claim of a fully verified\ndatabase engine? Well, the code *does* check if the hard-coded\nknowledge about the data generator holds in practice, e.g., by checking\nif the guessed join partner row is correct. If not, it falls back to a\nsuper slow generic nested loop implementation, but as that never happens\nwith the official TPC-H data generator, that code path is irrelevant in\nthe reported experiments.\n\nWhich brings me back to my original blog post claim: Letting AI agents optimize for a given database benchmark makes the benchmark useless. They will do their best to find insane tricks to speed up this particular benchmark, but none of that would work in a real system. The high tuning and synthesis time aside, deleting a single tuple (or inserting one in the middle) can most likely cause the whole approach to fall off a cliff, as it forces the slow fallback path. Which is not something you want in a production system.", "url": "https://wpnews.pro/news/database-architects-next-level-benchmarking-overfitting", "canonical_source": "http://databasearchitects.blogspot.com/2026/11/next-level-benchmarking-overfitting.html", "published_at": "2026-11-10 01:58:51+00:00", "updated_at": "2026-11-10 02:22:39.423652+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-research", "ai-safety"], "entities": ["GenDB-V", "Umbra", "DuckDB", "TPC-H", "TPC-D", "Database Architects", "GenLang-V"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/database-architects-next-level-benchmarking-overfitting", "markdown": "https://wpnews.pro/news/database-architects-next-level-benchmarking-overfitting.md", "text": "https://wpnews.pro/news/database-architects-next-level-benchmarking-overfitting.txt", "jsonld": "https://wpnews.pro/news/database-architects-next-level-benchmarking-overfitting.jsonld"}}