General methods that scale with computation will inevitably displace hand-engineered domain knowledge! Sutton’s Bitter Lesson hangs like a Sword of Damocles over all of us. This paper, which just dropped, applies that lesson to data agents, and says that as LLMs improve, they will quickly absorb the agent scaffolding researchers have spent the last few years painstakingly building. It argues that researchers should instead work on building curated contextual information about the data environment (aka. persistent semantic context) to help data agents be efficient across many queries.
The evaluation section aims to capture the Bitter Lesson in action. The authors compared general coding agents (the Codex harness with no task-specific engineering) against state-of-the-art human-designed data agents on two benchmarks, TAG-Bench and DAB. They use the same models on both sides, so the only variable is the scaffolding.
They find that:
Side Remark: Interestingly, the authors use these efficiency gains to bury their own prior work (in this case Ion and Matei's) from just a year ago. "Supporting Our AI Overlords" argued database systems would be overwhelmed by "agentic speculation": massive bursts of inefficient queries from confused models. Now they say the opposite, that models formulating correct answers efficiently "directly challenges the premise of recent work [11]". Them are fighting words. Why bury your own work when others would happily do it for you? For the record, I still think the direction in the Overlords paper is valid, and we should work on designing data systems for bursty AI workloads. Cheaper per query is not the same as less work for the database. This is Jevons paradox. If a query costs a fraction of what it used to, we will point many more agents at the data, running longer tasks, in parallel, in the background. Secondly, the main point of the Overlords paper was that queries are varied, yet our databases are built for repetitive workloads. Fewer turns per query does not make the queries look any more alike, so the load still arrives in bursts, and data systems will still need to be designed for it.
Ok, those efficiency gains are splendid, but they also expose new bottlenecks. Schema exploration increases from 16% of the turns with o3 to 25% with GPT-5.6 Sol. While it shrank in absolute terms, it now becomes the biggest remaining slice. The failure analysis also says the same thing. Once execution and coding errors are largely eliminated, over 60% of the remaining failures for GPT-5.6 Sol are semantic mistakes, such as not knowing the organization's idiosyncratic business definitions, join keys, or undocumented schemas. The model got smarter, but it still doesn't know about the intricacies of your data environment.
This motivates their research agenda proposal, which they test at small scale. When the authors let the AI "self-curate" a persistent context document by exploring the databases and 12 sample queries, and injected that text into the initial prompt, agent accuracy jumped by up to 19 percentage points.
But these improvements come with a price tag. Table 2 shows the cost of building these contexts, and some of them are very expensive. In their tiny setting (just 12 sample queries and 12 datasets) the schema-focused context took nearly an hour (3,400 seconds) and $9.60 to build, and GEPA cost $12.16. If you extrapolate that to an enterprise environment with terabytes of data and thousands of tables, the overhead becomes astronomical.
To address these overheads and the model's lack of environmental knowledge, the authors propose building a persistent semantic context layer. This context would be built offline, stored by the database system, and served to the agent to prevent redundant exploration. Section 3 outlines this agenda, dividing the problem into maintaining semantic consistency and designing efficient physical data structures.
Unfortunately, this section is the weakest part of the paper. Even when taking into account this is a position paper, the proposal remains superficial. It lists high-level categories like "consistency scope" or "consistency models" without offering technical solutions or even analysis.
There may also be a deeper problem here. The paper itself shows that agents can author their own context offline. If we take the Bitter Lesson to heart, why do we need the database community to build the semantic context layer? With improved models, agents will likely figure out their own memory formats, how to keep them consistent, and how to lay them out on disk, which is all of Section 3. No?
I actually like this problem a lot. The problem is real, and the economics get worse the bigger the organization. The same thing shows up one level above, in software engineering. Here hundreds of engineers point models at a large existing codebase, and every session spends time/money again to rediscover the conventions and ownership boundaries the organization already knows. The costs become prohibitive quickly.
The paper outlines several architectural directions for implementing this persistent semantic context layer natively within future data systems. One approach is to build the layer as a dependency graph, where business definitions and schema rules are linked as traceable nodes. Another option is treating the semantic layer like a live materialized view, using database events to trigger targeted AI rewrites whenever the underlying data shifts. The idea is to shift AI memory from a static text file into a metadata-driven component of the database.
Of course, I have my thoughts and biases on this. I think TLA+ deserves serious consideration at this semantic context layer. As Boris Cherny recently highlighted, formal specification/verification languages like TLA+ should no longer be seen as niche. He showed how he uses Opus 5.5 to verify a codebase, finding hidden bugs and race conditions with TLA+ in just a few short prompts. These models are now fluent enough in TLA+ that you do not need to be an expert in the language to get value out of it.
While TLA+ is most famous for checking the interleaving and ordering of events in distributed systems, at its core it is just set theory plus temporal logic. That makes it versatile. It can model schema properties, define strict relations between data facts, and map out structural dependencies. More importantly, it enables you to capture not just static safety invariants, but also the temporal properties the system must guarantee over time. This flexibility makes TLA+ a compelling answer to the paper's consistency problem. Instead of treating the persistent semantic context as Markdown text, an AI agent could continuously translate natural-language business rules into explicit TLA+ relations. If an underlying data-access protocol is modified or a core business metric is redefined, the system can use the TLA+ model to show exactly which downstream semantic dependencies break. Beyond keeping memory consistent, this formal foundation also helps the agent answer queries. Before executing a complex analytical query, the agent can check its proposed logic against the TLA+ invariants to rule out superfluous join paths. For example, it would immediately catch that querying for records where a "delivery timestamp" precedes an "order timestamp" violates a temporal invariant, and fix its filtering logic before ever touching the database.
I hate that the Bitter Lesson is so real, and so bitter. However, as I noted in my recent post on LLMs, these models excel at "high-throughput mediocrity". The Bitter Lesson applies best where statistical approximation is acceptable and the domain is flexible. When you need absolute correctness and performance, I still think human-engineered precision can hold the line. I still believe in the importance of good algorithms and abstractions. So, even if it proves futile, I keep looking for ways to push back against the Bitter Lesson, partly as an act of defiance, and partly as my sworn duty as a loyal member of the systems craft club. Grrr.
Which brings me back to the title: What happens when the model eats the stack?
I guess it takes a massive core dump!
Ba dum tss... I'll see myself out.
PS: My colleague Jesse's review of the paper is also worth reading.