Article: AI Changes the Data Interface, Not the Query Engine

A disaster-recovery analysis prototype that stored thirty-second service metrics as Parquet in S3-compatible object storage and used DuckDB for deterministic aggregation worked without a shared lakehouse, but only because the query engine became an embedded component inside the AI agent workflow rather than disappearing. The architecture separates deterministic computation from probabilistic reasoning: engines filter, aggregate and join while agents interpret evidence and communicate conclusions, with agent-generated SQL validated against an allowlist of datasets and operations and executed in a sandboxed runtime. An embedded engine such as DuckDB suffices when data volume is moderate, the primary consumer is the agent workflow and writes are controlled, while a managed lakehouse is justified by multiple independent consumers, joins with large shared datasets, concurrent writes or organization-wide governance.

Key Takeaways - An AI agent that reads Parquet still depends on an execution engine; AI changes the analytical interface and interpretation layer, not the need for precise computation. - Separate deterministic computation from probabilistic reasoning. Engines should filter, aggregate, and join, while agents should interpret evidence and communicate conclusions. - An embedded engine such as DuckDB can be sufficient when the data volume is moderate, the primary consumer is the agent workflow, and writes are controlled. - A managed lakehouse becomes valuable when data has multiple independent consumers, must join with large shared datasets, requires concurrent writes, or needs organization-wide governance. - Production-grade agent analytics should retain the source data, configuration, derived facts, and model outputs so that every report can be validated and replayed. AI agents can now generate analytical code, read files from object storage, and turn operational data into structured reports. Does that make data lakes, lakehouses, and SQL engines obsolete? We encountered this question while automating the analysis of a large disaster-recovery exercise. Our first working design stored thirty-second service metrics as Parquet in S3-compatible object storage, used DuckDB for deterministic aggregation, and asked an AI agent to classify failover outcomes and explain the evidence. The prototype worked without a shared lakehouse, but not because the query engine disappeared. Instead, the engine became an embedded component inside the agent workflow. This article explains the architecture, introduces a practical separation between deterministic computation and probabilistic reasoning, and presents four signals for deciding when an embedded engine is sufficient and when a managed lakehouse is justified. The Question That Appeared After the Prototype Worked The question did not come from a whiteboard exercise. It appeared after a real workflow successfully produced its first report. We were analyzing a disaster-recovery DR islanding exercise. During the exercise, one region was intentionally isolated, and traffic was expected to fail over to surviving regions. For every service, we needed to answer three questions: - Did the surviving region absorb the traffic that left the isolated region? - After absorbing the traffic, did the service remain available? - Did latency remain within an acceptable range, or did the additional load create sustained degradation? We exported service metrics at thirty-second resolution, wrote them as Parquet files to S3-compatible object storage, and let an AI agent generate a structured analysis report. Once the workflow ran end to end, an attractive idea emerged: If the agent can read files from object storage and produce the answer, why do we need tables, a catalog, a lakehouse, or SQL at all? The question sounds reasonable because the user-facing experience has changed. Instead of opening a dashboard or writing a query, an engineer asks the agent to analyze the exercise. The agent identifies the relevant data, generates DuckDB SQL or Python as needed, executes the analysis through deterministic tools, and explains what happened. For example, to verify whether traffic shifted from Region A to Region B during failover, the agent may generate a read-only DuckDB query over a predefined set of tables and time ranges. Before execution, the query is validated against an allowlist of accessible datasets and operations, then executed in a sandboxed analysis runtime. The runtime returns structured query results, which deterministic application code normalizes into a fact packet containing fields such as traffic volume by region, timestamps, and percentage changes. A simplified implementation looks like this query: generated sql = """ SELECT region, SUM request count AS requests FROM traffic metrics WHERE ts BETWEEN :start time AND :end time GROUP BY region """ validate sql generated sql, allowed tables={"traffic metrics"}, allowed operations={"SELECT"}, rows = sandbox.execute generated sql, params={ "start time": exercise start, "end time": exercise end, }, fact packet = { "metric": "request count", "window": exercise start, exercise end , "by region": { row "region" : row "requests" for row in rows }, } In this simplified example, the agent generates only the analytical query. A deterministic validation layer rejects access to non-allowlisted tables or non-read-only operations before execution. The query then runs in a sandboxed runtime and the returned rows are converted by application code into a structured fact packet. The model receives this packet for example, request counts for Regions A and B rather than inventing or directly estimating those values. Why DR Analysis Is More Than Scanning Metrics The difficult part of DR analysis is not retrieving a time series. It is deciding what the time series means. Suppose Region A is isolated and Region B remains active. A healthy failover should produce a recognizable sequence: - QPS decreases in Region A. - QPS increases in Region B by a corresponding amount. - The success rate in Region B remains above the service objective. - Tail latency may rise briefly, but it should not remain elevated. Each statement requires context. A QPS increase is not automatically a successful failover. It may be too small to account for the traffic lost in the isolated region. A lower success rate is not automatically a DR failure if the metric includes expected business rejections. A latency spike at the cutover boundary may be harmless if the system quickly returns to a steady state. We therefore used two baselines and a steady-state window: - Historical baseline. the same time window on the previous day, which captures normal time-of-day behavior. - Immediate baseline, the hour before the exercise, which captures the most recent service state. - Steady-state window, a post-cutover interval that excludes the short transition period and focuses on sustained behavior. The analysis collected four core signal families: - qps - err qps - success rate - latency pNN, including tail latency such as p99 For each service, deterministic code calculated facts such as traffic absorption ratio, SLO violations, latency change, and the duration of sustained degradation. The agent then interpreted those facts in the context of the service, the exercise configuration, and the available evidence. This distinction is fundamental. The hard part is often the judgment, but the judgment is only trustworthy when the facts underneath it are deterministic. Architecture: Object Storage, an Embedded Engine, and an Agent The workflow used three logical layers: data preparation, deterministic computation, and agent-driven interpretation, with reproducibility treated as a cross-cutting requirement. The data-preparation layer used reusable DuckDB build logic, which was generated once and then reused, to transform raw metrics into self-describing assets in object storage. For each analysis request, the agent selected the relevant inputs, generated constrained SQL or Python, and orchestrated deterministic execution in DuckDB/Python. The resulting structured fact packets were then interpreted by the agent to produce the final report. Throughout the workflow, the input data, configuration, reusable build logic, deterministic facts, and agent execution metadata were retained so that the analysis could later be reconstructed, replayed, and evaluated. Figure 1. The AI agent initiates and orchestrates the analysis, deterministic tools compute the facts, and the agent interprets the resulting fact packet. Source: Authors’ own illustration, created with assistance from ChatGPT. Storage and Self-Description The physical data lived in S3-compatible object storage. Each exercise had a self-describing directory: s3://<bucket /<root-prefix / index.json <exercise-id / manifest.json exercise config.json metrics- .parquet service dimension.parquet The files served different purposes: - index.json listed available exercises. - manifest.json recorded the schema, row counts, file sizes, and checksums. - exercise config.json captured time windows, regions, thresholds, and exercise identity. - metrics- .parquet contained the thirty-second metric facts. - service dimension.parquet contained service metadata such as ownership and service tier. The manifest was important because a folder of files is not yet a trustworthy data asset. Consumers need to know what is present, which schema was used, whether the write completed, and whether the files can be validated later. Deterministic Processing with DuckDB DuckDB https://duckdb.org/why duckdb was used on both sides of the storage boundary. During ingestion, it read the raw metric records, sorted them, wrote compressed Parquet files, and generated validation statistics for the manifest. During analysis, it read the Parquet files directly, filtered the relevant windows, joined service metadata, and computed the fact packet used by the agent. This architecture benefited from several properties of DuckDB: - It runs in-process, so it can travel with the agent workflow instead of requiring a separate database service. - It reads Parquet directly and can push filters and column selection into Parquet scans. - It provides SQL semantics for aggregation and joins. - It can spill larger operations to local storage instead of requiring the entire working set to remain in memory. The point was not that DuckDB is universally better than Spark https://aws.amazon.com/what-is/apache-spark/ , Trino https://www.cloudera.com/resources/faqs/what-is-trino.html , or a managed lakehouse. The point was that the workload did not yet need a shared distributed engine. Probabilistic Interpretation with the AI Agent The agent did not receive hundreds of thousands of raw metric rows and attempted to count errors or average latency itself. Instead, it received a compact, deterministic evidence packet for each service. A simplified packet looked like this example: { "service id": "service-a", "exercise window": "post-cutover-steady-state", "expected failover qps": 1200, "observed qps increase": 910, "traffic absorption ratio": 0.758, "success rate baseline": 0.9992, "success rate during exercise": 0.9941, "service slo": 0.9990, "p99 latency baseline ms": 84, "p99 latency during exercise ms": 191, "sustained degradation minutes": 18, "data quality flags": } The agent used the packet to classify the outcome, explain the supporting evidence, and propose the next investigation step. A corresponding output might be: { "classification": "partial failover", "confidence": "high", "evidence": "The surviving region absorbed approximately 76% of the expected traffic.", "Availability remained below the service SLO during the steady-state window.", "The p99 latency increase was sustained for 18 minutes." , "recommended action": "Review routing capacity and downstream dependency limits." } The exact numbers came from the engine. The model's responsibility was to determine how those numbers should be interpreted and communicated. A simple threshold-based rule was not always sufficient. For example, a service might temporarily exceed its p99 latency threshold immediately after cutover while traffic had already shifted correctly and availability remained within the service objective. A rule that treated any threshold violation as a failure would classify the service as degraded. The agent could instead compare the transition window with the steady-state window, recognize that latency returned to baseline within a few minutes, and classify the event as a successful failover with transient degradation rather than a sustained DR failure. More importantly, some cases contained conflicting evidence that could not be resolved by a single threshold or temporal rule. A service might absorb the expected traffic and recover its own latency, yet still remain at risk because a downstream dependency was operating near saturation. Conversely, a Tier-2 asynchronous service could show a temporary backlog increase without violating its availability objective, while the same pattern in a Tier-0 synchronous dependency would warrant escalation. In these cases, the deterministic layer still produced the underlying facts i.e.,traffic shift, latency, success rate, backlog, dependency saturation, service tier, and SLO status , but the agent combined those facts with service context to determine which evidence was most important and how the event should be classified. The agent therefore added value not by replacing deterministic computation, but by resolving context-dependent and sometimes conflicting evidence that would otherwise require an increasingly large set of service-specific rules. The Determinism Split The most reusable lesson from the project is what I call the determinism split: separate exact computation from contextual reasoning. | Give to the engine | Give to the AI agent | | Filtering, aggregation, joins, sorting, and window calculations | Selecting relevant evidence and explaining why it matters | | Exact and reproducible numerical results | Classifying failover behavior and suggesting investigation paths | | Operations whose cost grows with data volume | Work that depends on context, trade-offs, and communication | | Data quality checks and validation flags | Abstaining when the available evidence is insufficient | Table 1. A practical separation between exact computation and contextual reasoning. This division avoids a common anti-pattern, using an LLM as a query engine. A language model should not read hundreds of thousands of rows and calculate an error count, percentile, or average. That approach is slow, expensive in tokens, and difficult to reproduce. More importantly, it assigns deterministic work to a probabilistic system. The safer pattern is: - Use a deterministic engine to produce a small set of authoritative facts. - Allow the agent to generate constrained SQL or Python needed to derive those facts. - Execute that code through deterministic tools and give the resulting fact packet to the model with the necessary context. - Require the model to explain conclusions using fields from the fact packet, while retaining inputs, generated code, execution metadata, and outputs for replay and audit. The agent can decide which question to investigate, dynamically generate constrained DuckDB SQL or Python, and orchestrate its execution. The generated code may be proposed probabilistically, but the resulting numerical facts must come from deterministic execution rather than from the model itself. What the Workload Told Us About the Architecture The exercise dataset contained: - Thirty-second metric resolution - 1,127,748 fact rows - 1,523 services - Approximately 3.8 megabytes of compressed Parquet fact data The upstream collection run across all services took roughly an hour with concurrent workers. The resulting analytical dataset was compact enough that an embedded engine could process it close to the agent without introducing a shared data platform. That observation matters because architecture decisions should follow the actual workload rather than the label big data. More than a million rows can sound large in a design review, but row count alone is not a sufficient reason to deploy a distributed lakehouse stack. Column count, compression, query shape, concurrency, access patterns, and data movement all matter. A Representative Analysis Walkthrough The following is a representative example based on the workflow used in the exercise; service identifiers and values have been anonymized. For one service, the deterministic execution layer produced a fact packet showing that the surviving region absorbed approximately seventy-six percent of the expected failover traffic, availability fell below the service SLO during the steady-state window, and p99 latency remained elevated for eighteen minutes. At the same time, the service continued to serve traffic and did not experience a complete outage. The agent interpreted these facts as a partial failover rather than a binary success or failure. Its explanation cited the traffic absorption ratio, the sustained SLO violation, and the latency degradation, and recommended reviewing routing capacity and downstream dependency limits. The conclusion was then validated against the underlying deterministic facts and the exercise timeline. Reviewers could trace each statement in the agent’s explanation back to fields in the fact packet and rerun the same DuckDB/Python analysis against the retained input data and configuration. Once the conclusion was confirmed, the resulting action was to investigate whether routing capacity or a downstream bottleneck had prevented the surviving region from absorbing the full expected load. This validation step was important: the agent proposed the interpretation and next action, but the numerical evidence remained reproducible and independently verifiable. Making the Workflow Reproducible The prototype only became an engineering system after we treated reproducibility as a first-class requirement. For every exercise, we retained a replay bundle containing: - The Parquet inputs - The manifest and checksums - The exercise configuration - The generated SQL or Python code - The execution metadata - The deterministic derived facts - The model output - The agent version and prompt - The model identifier This replay bundle made it possible to rerun an analysis offline, investigate a disputed conclusion, or compare the behavior of two agent versions against the same evidence. We also encoded data semantics explicitly. One subtle example was the meaning of a missing value. Missing QPS or error-count data could represent zero activity in some contexts. Missing success-rate or latency data often meant no traffic was available to measure, not zero percent success or zero milliseconds of latency. Filling every missing value with zero would have created false failures and false recoveries. The schema and data-quality flags needed to preserve that distinction so the engine and the agent interpreted the dataset consistently. When the Lightweight Architecture Stops Being Enough Agent plus DuckDB is not the final architecture for every analytical system. It works when the workload has the right shape. Four signals indicate where the lightweight architecture begins to encounter pressure. They do not require adopting a complete lakehouse stack at once. Each signal points to a different capability that can be introduced incrementally, including a catalog for shared discovery and governance, an open table format for safe concurrent writes and evolution, or a distributed query engine when computation must move closer to large shared datasets. Independent Consumers Appear If dashboards, scheduled jobs, notebooks, or other teams need direct access, the data needs a stable contract. Those consumers should not have to call the original agent or reverse-engineer a bucket layout. A shared catalog allows them to discover a table by name, understand its schema, apply permissions, and query it through standard tools. The Data Becomes Too Large to Move to the Agent Environment At terabyte scale, pulling files toward a local or development environment for analysis becomes inefficient. The query should execute where the data is stored i.e., near-data processing https://www.emergentmind.com/topics/near-data-processing-ndp , with partition pruning, predicate pushdown, and distributed processing. Only the result should move back to the agent. The Workflow Must Join with Large Shared Datasets DuckDB can join local files efficiently when both sides are small enough. The boundary appears when the other side is an organization-wide authoritative dataset that is already registered in a shared catalog and is too large or too frequently updated to export safely. At that point, it is more reliable to register the new data in the same environment and execute the join through the shared engine. Concurrent Writes or Table-Level Evolution Are Required A single-writer convention is simple and effective for an early workflow. It breaks down when multiple producers update the dataset, schemas evolve independently, or readers require consistent snapshots. An open table format such as Apache Iceberg https://aws.amazon.com/what-is/apache-iceberg/ adds table metadata, snapshots, schema evolution, and transactional commit semantics on top of the Parquet files. A managed lakehouse can then combine three capabilities: - A catalog for discovery, ownership, permissions, schema, and partitions - An open table format for snapshots, evolution, and safe writes - A shared query engine for distributed scans and joins These capabilities are valuable, but they introduce integration, governance, compute, and operational cost. They should be adopted in response to real sharing, scale, and transaction requirements, not because every AI project is assumed to need a lakehouse on day one. A Practical Decision Table | Workload signal | Embedded engine close to the agent | Managed lakehouse | | Primary consumer is one agent workflow | Strong fit | Often unnecessary | | Moderate, compressed analytical dataset | Strong fit | May be over-engineered | | Ad hoc or exercise-based analysis | Strong fit | Useful only if sharing is required | | Many independent consumers | Weak fit | Strong fit | | Terabyte-scale scans | Weak fit | Strong fit | | Joins with large shared authoritative tables | Limited | Strong fit | | Concurrent writers and snapshot requirements | Weak fit | Strong fit | | Organization-wide catalog and governance | Limited | Strong fit | Table 2. Choose the platform based on access patterns, scale, and write semantics. The decision is not AI or SQL nor is it files or tables. The real decision is: Is an embedded, agent-driven analytical engine sufficient, or does the workload require a managed, shared data platform? Conclusion: AI Changes the Interface, Not the Engine AI agents make data analysis more accessible. They can translate an operational question into analytical steps, select relevant evidence, classify behavior, and write a report that an engineer can act on. They do not eliminate the need for precise execution. In our DR analysis workflow, Parquet and DuckDB performed the mechanical work: storage, filtering, aggregation, validation, and joins. The agent performed the contextual work by deciding how to interpret traffic absorption, availability, and latency together, and communicating the result. The architecture remained intentionally lightweight because the real workload did not yet require a shared lakehouse. What changed after operating the workflow was our view of where the analytical interface should live. Engineers no longer needed to interact directly with the query engine for every investigation, but the engine remained essential for producing trustworthy facts. The agent became the layer that planned the analysis, generated the required SQL or Python, and interpreted the evidence. The main architectural trade-off is therefore not AI versus SQL, or files versus tables. It is how much analytical infrastructure needs to sit behind the agent for the workload at hand. A small embedded engine may be sufficient for a focused workflow; shared catalogs, open table formats, and distributed engines should be introduced only as sharing, scale, write concurrency, or governance create a concrete need. The engine remains; the interface changes.