An AI data agent turns a plain English question into a query, runs it, and hands back an answer. The demo is always impressive. The part that decides whether it survives contact with a real company is not the query generation at all, it is what happens on either side of it.
Most teams evaluating these tools grade them on the wrong thing. They ask whether the agent got the SQL right on a question they already knew the answer to. The question that matters is whether they would forward the agent's answer to someone who is about to spend money on it.
A capable model turns "which campaign drove the most revenue last quarter, broken down by channel" into a join across campaign and revenue tables, filtered to a date range, grouped by channel. That has been reliable for a while now, and it is the part every vendor demos.
What it does not tell you is whether the revenue table double counts refunds, whether the campaign table has rows with a null channel that silently drop out of the join, or whether last quarter is missing three days because an ingestion job failed on a holiday weekend.
The query was correct. The answer was wrong. Nothing in the SQL tells you which one you got.
An agent that gets used past the pilot does three things the demo version does not.
It profiles the data before it analyzes it. Row counts, null rates, duplicate keys, date coverage. If a column that should be complete is forty percent null, that is worth saying out loud before producing a chart from it.
It shows the path it took. Which tables, which join keys, which filters, and why. Not a full query dump for a business user, but enough that someone who knows the schema can audit the reasoning in under a minute.
It refuses to conclude when the data will not support a conclusion. Saying "this segment has eleven rows, I would not read a trend into it" is the single most valuable behavior a data agent has, and it is the one most likely to be tuned out in favor of an answer that looks confident.
The failures in production are rarely dramatic. They are quiet and directional.
An agent picks the wrong one of two similarly named tables, one of which is a stale copy. It applies a filter the user implied but did not state. It joins on an ID that is unique in one table and not in the other, quietly inflating a total. It answers a question about "customers" using a table that counts accounts.
Every one of those produces a number that looks completely reasonable. That is the whole problem. A wrong answer that looks wrong gets caught. A wrong answer that looks right gets used.
This is why an agent that answers fast and wrong is worse than a dashboard nobody opens. The dashboard's staleness is at least visible.
Buying gets you connectors, a working interface, and someone else maintaining the integrations. For a team that mostly needs faster answers on a standard warehouse, that is the right call and it is not close.
Building makes sense when you need control over what the agent is allowed to conclude, how it reports uncertainty, and where the data is allowed to travel. Those constraints are hard to bolt onto a product designed to always produce an answer, and they get more important as the answers start moving budget.
The honest middle path is to buy first, find out which of your questions the agent gets quietly wrong, and let that list tell you whether you have a build problem or a data modeling problem. Most teams discover it is the second one.
Judge a data agent on what it does when the data is bad, not on what it does when the data is clean. Ask it a question your warehouse cannot actually answer and watch what comes back. If it produces a confident chart anyway, you have learned everything you need to know.
There is a fuller breakdown of how these systems work under the hood, the current landscape, integration patterns, and the accuracy and trust problem in this guide to AI data agents.