# AI Data Agent Guardrails: How to Keep Conversational Analytics From Inventing Business Metrics > Source: > Published: 2026-09-17 20:01:01+00:00 *Conversational analytics is becoming a first stop for business questions. The hard part is making sure its answers mean the same thing as the numbers Finance, Product, and Sales already trust.* Someone asks an AI data agent, “Why did revenue fall last month?” The answer sounds sharp. It names a segment, suggests a cause, and ends with a tidy recommendation. Then Finance checks the dashboard and gets a different number. This is not usually a model-intelligence problem. It is a meaning problem. The agent may have counted bookings where Finance counts recognized revenue, used the calendar month instead of the fiscal month, or followed an old join that quietly double-counts a source. Fluent prose can hide all of that. The timing matters. OpenAI recently [introduced a Data agent for ChatGPT Work](https://openai.com/index/put-data-to-work/) that can investigate company data and create shareable dashboards. Its earlier [account of the in-house data agent](https://openai.com/index/inside-our-in-house-data-agent/) makes the right operating principle clear: the agent should inherit existing permissions, surface assumptions, and link people back to the underlying results. That is the bar worth designing for, regardless of whether you use ChatGPT Work, Databricks Genie, Power BI, Looker, a warehouse assistant, or a home-built text-to-SQL stack. The practical answer is an **AI data agent guardrail**: a set of machine-readable rules that turn a vague request into a governed metric query and return proof with the answer. This guide shows how to build that layer without waiting for a perfect semantic program. A dashboard answers a small number of predesigned questions. An agent can explore. That is why it feels useful. It can move from “revenue fell” to “which region changed, which channel changed, and what changed inside that channel?” in one conversation. But exploration creates three new failure paths: Recent practitioner discussions make this concrete. Teams report that a semantic layer helps, but they also worry about upstream schema changes that leave a technically correct metric producing the wrong business answer. Others describe dashboards shifting into a health-check role: a person sees a red signal, asks an agent to investigate, then verifies the answer against the evidence layer. That is a healthier frame than declaring dashboards dead. Your aim is not to make the agent sound certain. It is to make uncertainty visible and routine verification cheap. A metric contract is a compact, versioned agreement about one business number. It gives an agent more useful context than a warehouse schema and gives reviewers a stable thing to audit. It can live in dbt metadata, a semantic-layer tool, a repository, or a small service. The storage choice matters less than the contract being canonical and testable. For a first rollout, select five to ten high-value metrics that people ask for repeatedly. Good candidates are net revenue, active customers, conversion rate, retained accounts, support backlog, and qualified pipeline. Avoid “every table” as the first milestone. Each contract should answer six questions: Here is a deliberately small example. The point is not YAML. The point is that the agent receives a constrained interface instead of raw tables plus a hopeful prompt. ``` metric: net_revenueversion: 3owner: finance-analyticssource: mart.finance_monthlyexpression: sum(recognized_revenue_usd) - sum(refunds_usd)grain: month, legal_entity, regioncalendar: fiscal_445timezone: America/New_Yorkallowed_filters: - region - legal_entity - product_familyrequired_evidence: - metric_version - generated_sql - source_freshness - applied_filters - result_linkconfidence_rules: stale_after_hours: 30 escalation_on_missing_dimension: true ``` Notice what is missing: a prose instruction telling the model to “be accurate.” A model cannot enforce that request by itself. A contract can limit what is queryable, reject an unsupported filter, and make the returned answer carry its own receipt. People do not ask for recognized_revenue_usd. They ask for revenue, ARR, new business, or expansion. Add approved aliases to the contract, plus a short explanation of terms that are commonly confused. If “revenue” is ambiguous in your company, do not guess. Make the agent ask: “Do you mean recognized revenue or contracted bookings?” That clarification can feel slower in a demo. In a planning meeting, it is much faster than discovering a decision was based on the wrong measure. Many early AI analytics projects connect a model directly to a database with read-only credentials. Read-only is necessary, but it is not enough. The database can still expose confusing joins, sensitive columns, unapproved calculations, and a huge surface for expensive scans. Instead, expose a narrow set of tools that operate on governed objects: This design works through an MCP server, a conventional API, a warehouse semantic layer, or a tool wrapper in your agent framework. It also creates a clean place to enforce row-level security. Pass through the requesting user’s identity when possible; do not give a general-purpose agent a broad service account and hope it remembers who should see what. *Let the model decide how to explain a governed result. Do not let it decide what your company’s revenue means.* The best AI analytics answer is not a paragraph. It is a paragraph plus enough context to inspect and reproduce the claim. Call that an evidence receipt. For low-stakes exploratory questions, the receipt can stay collapsed behind a “show work” control. For executive reporting, finance, pricing, customer commitments, or automated alerts, show it by default. A useful receipt includes: That last point is important. An agent should be allowed to say, “I can see a 14% decline in net revenue for the requested period, but the campaign-attribution feed is six hours late, so I cannot yet rank channels reliably.” A cautious incomplete answer is more useful than a complete-looking one that hides a broken dependency. Generic text-to-SQL benchmarks are not enough. They rarely include your fiscal calendar, duplicate customer identities, restricted account fields, or loaded words such as “customer,” “revenue,” and “active.” Build an evaluation bank from real analytics tickets, recurring executive questions, and past incidents. Each evaluation case should include the user question, authorized role, expected metric contract, allowed interpretations, a known-good result, required receipt fields, and a failure label. Start with about 50 cases. Add a case whenever someone catches an answer that is confusing, incorrect, overly broad, or missing proof. Test four types of behavior: Use deterministic assertions for numbers and permissions. Use human or model-assisted review for explanation quality, but never let a language-model judge be the only authority on whether the data query was correct. High usage can mean people love the agent. It can also mean they are repeatedly trying to get a trustworthy answer. Track a smaller set of operational measures: Do not punish clarification. A rising clarification rate early in the rollout may mean the agent is finally exposing ambiguity that dashboards had quietly buried. Choose one team with strong data ownership and a clear recurring decision loop. Product growth, customer success, or revenue operations often work well. Give the agent only curated metrics, read-only tools, and one approved output channel. A practical four-stage sequence looks like this: Keep a simple escalation lane. When the agent cannot map a question to a contract, encounters stale data, or produces a result that conflicts with a certified view, it should stop and hand off the question with the relevant context. “I do not know” is a feature when it comes with a useful next step. A metric contract can be perfectly written and still lead to a bad answer. Imagine that the definition of active users correctly counts unique user_id values. A new guest checkout flow then begins sending null IDs, or a partner begins replaying events. The query remains valid. The number becomes misleading. That is why the contract needs a few quality signals beside the definition: expected freshness, null-rate limits for critical fields, volume-change thresholds, and a link to the latest source tests. The agent does not need to diagnose every anomaly itself. It needs to know when it is unsafe to present a clean answer. For example, a query_metric response might carry a machine-readable warning: ``` { "metric": "active_customers", "value": 48231, "quality": "warning", "warnings": [ "Guest-session user_id null rate is above its agreed threshold", "The event source is 19 hours behind its normal refresh" ], "recommended_action": "Do not compare this period with the prior week until the source check passes."} ``` Make the agent repeat that warning in plain language, rather than burying it in a trace. This separates a useful exploratory answer from a decision-ready answer. It also gives data teams a focused backlog: improve the source contract, not another free-form prompt. Metrics change for good reasons. Finance may change a revenue-recognition rule. Product may split one event into two. Sales may redefine a qualified lead. A mature agent should not silently use the newest definition for a historical answer if the question is about a prior decision. Version the contract and record the effective date. When a definition changes, run the evaluation bank against both versions, publish a short change note, and decide how historical comparisons should behave. In some cases, restating history is correct. In others, preserving the old definition is necessary to explain what a team knew at the time. The key is to make the choice explicit. Suppose a customer-success lead asks, “Why did enterprise retention dip this quarter?” A safe agent flow looks like this: Now compare that with a raw text-to-SQL path. The model sees tables named accounts, subscriptions, and invoices. It guesses a retention formula, misses an exclusion for internal accounts, and creates a convincing explanation. The second path may be faster to prototype. The first is faster to trust. This is also where model choice becomes less central. A stronger model can improve planning and explanations. It cannot infer a company’s authoritative retention definition from column names. Better context is not merely extra tokens; it is a constrained agreement about meaning and permission. Prompts are helpful explanations, not a source of truth. They drift, get truncated, and are hard to audit. Point the agent to a versioned metric contract or tool instead. This defeats the access model that made your warehouse safe. Use pass-through identity or a broker that applies the user’s entitlements at query time. Do not force everyone to read SQL. Do make the underlying result and definitions one click away for the people accountable for decisions. Start with descriptive and comparative questions. Causal claims usually need experiment context, external events, and domain judgment that a metric contract alone cannot provide. AI data agents will make it easier for more people to ask useful questions. They will also reveal where your organization has several answers to the same question. That can be uncomfortable, but it is productive. The teams that earn trust will not be the ones with the most polished analytics chatbot. They will be the ones that make definitions explicit, keep access scoped, test real questions, and attach evidence to every important claim. Build those habits now, and a new model or data-agent platform becomes a swap in the interface layer rather than a new governance emergency. They are technical and operational controls that keep an AI analytics agent within approved metric definitions, permissions, data scopes, freshness rules, and evidence requirements. You need a canonical governed interface, but it does not have to begin as a large commercial semantic-layer deployment. A small versioned set of metric contracts and restricted tools is a useful first step. Do not let them generate calculations against arbitrary raw tables. Route requests to approved metrics, validate supported filters, and return the query result and assumptions with the answer. It can, but raw free-form SQL should be constrained and reviewed in high-risk environments. For common business questions, metric-level tools or compiled queries are safer and easier to evaluate. Include the metric version, source freshness, time period, filters, query or result link, assumptions, and warnings that could change how a reader interprets the answer. Start with five to ten important, well-owned metrics. Expand after the team can explain disagreements, maintain evaluation cases, and update contracts when definitions or source data change. [AI Data Agent Guardrails: How to Keep Conversational Analytics From Inventing Business Metrics](https://pub.towardsai.net/ai-data-agent-guardrails-how-to-keep-conversational-analytics-from-inventing-business-metrics-df1dc7f82a6c) was originally published in [Towards AI](https://pub.towardsai.net) on Medium, where people are continuing the conversation by highlighting and responding to this story.