{"slug": "your-ai-might-be-fine-but-your-data-is-lying-to-it", "title": "Your AI might be fine but your data is lying to it", "summary": "A developer argues that AI agents acting on enterprise data face a different reliability problem than human analysts: stale or ambiguous data that once produced merely bad analysis can now trigger bad operational actions, such as a procurement agent reordering inventory based on a copy of the data that has not yet caught up with a recent delivery. Citing a small benchmark, the developer notes that pre-modeled business definitions in a semantic layer improved accuracy and, more importantly, admitted when it could not answer, whereas a model writing SQL directly returned plausible but wrong numbers. The developer concludes that latency, availability, concurrency, and workload isolation now make analytical queries look more like production application infrastructure than traditional analytics.", "body_md": "# **Stale Data Used to Be Annoying**\n\nImagine a procurement agent checking inventory and deciding that stock has fallen below the reorder threshold. It places another order. The problem is that a large delivery was recorded a few minutes earlier, and the copy of the data the agent queried has not caught up. A dashboard [running on stale data](https://www.confluent.co.uk/blog/2026-q3-confluent-intelligence-ai-update/?utm_source=gradientflow&utm_medium=newsletter) might show you the wrong inventory number. An agent running on stale data can place the wrong order.\n\nFor years, many analytical systems could tolerate some delay, ambiguity, and imperfect data because a person sat between the result and the action. That person could wait for a slow query, notice that a number looked wrong, or ask which definition applied. Agents remove much of that buffer. They [check the data, take an action, see what happened](https://aws.amazon.com/blogs/big-data/powering-agentic-ai-with-real-time-streaming-data-on-aws/?utm_source=gradientflow&utm_medium=newsletter), and check again.\n\n[Last week I wrote](https://gradientflow.substack.com/p/what-counts-as-valuable-company-data?r=ks4p&utm_campaign=post&utm_medium=web&showWelcomeOnShare=true) about the value of preserving the history of how work gets done. This week I want to look at what an agent needs to know while it’s carrying out a task..\n\n#### **Relevant Is No Longer Enough**\n\nThe first wave of enterprise AI focused heavily on retrieval: find the information relevant to a question and put it in front of the model. Agents need to answer a different question. Not just what is relevant, but [what is true right now](https://docs.confluent.io/cloud/current/topics/real-time-context-engine.html?utm_source=gradientflow&utm_medium=newsletter). Did the payment clear? Is that inventory still there? Was the account suspended an hour ago? Did another process already touch the record? Information that is five minutes old may be fine for a research question and dangerously wrong for an action.\n\nFreshness is not the only problem. Definitions matter too. Suppose two systems disagree about what counts as an active customer. An analyst who notices the ambiguity can stop and ask somebody. An agent may choose one definition and keep going.\n\nA [small benchmark](https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026?utm_source=gradientflow&utm_medium=newsletter) illustrates the problem. When business definitions and relationships were modeled ahead of time, accuracy improved substantially. I find the failure mode more interesting than the scores. When the semantic layer, which encodes those definitions and relationships, could not answer a question, it said so. When the model wrote SQL directly against the data, it could return a [perfectly plausible wrong number](https://docs.getdbt.com/blog/semantic-layer-vs-text-to-sql-2026?utm_source=gradientflow&utm_medium=newsletter).\n\nWhen data informed people, ambiguity produced bad analysis. When agents act on that data, the same ambiguity can produce bad operations.\n\n#### **Analytical Queries Become Production Dependencies**\n\nCorrect answers are not enough if they arrive too slowly or not at all. A human running an analytical query can usually wait. An agent may be waiting on that query before it takes the next step.\n\nIf a query is step seven of twenty, every extra second can delay the steps behind it. A timeout can stop a customer interaction or business process rather than merely leave someone staring at a dashboard. Software can also launch many tasks at once and revisit the same data as conditions change.\n\nThat puts different expectations on some analytical systems. [Latency, availability, concurrency](https://aws.amazon.com/blogs/big-data/powering-agentic-ai-with-real-time-streaming-data-on-aws/?utm_source=gradientflow&utm_medium=newsletter), and workload isolation start to matter in ways that look more like application infrastructure than traditional analytics.\n\n#### **Data Quality Matters More When AI Acts**\n\nStale records, duplicates, inconsistent definitions, and poorly documented fields are [familiar data problems](https://www.oreilly.com/radar/the-quest-for-high-quality-data/?utm_source=gradientflow&utm_medium=newsletter). Better models do not make them disappear. What is different is what can happen when an agent acts on them.\n\nA duplicate record can trigger duplicate outreach or a second transaction. A bad [entity match](https://gradientflow.substack.com/i/129531113/entity-resolution-insights-and-implications-for-ai-applications) can cause an action against the wrong account. A poorly documented field can quietly change what an agent thinks it is measuring.\n\nIn one experiment, adding profiling and query history left an analytics agent [around 40% reliable](https://roundup.getdbt.com/p/the-context-engineering-playbook?utm_source=gradientflow&utm_medium=newsletter). Cleaning up the [data model and improving the documentation](https://roundup.getdbt.com/p/the-context-engineering-playbook?utm_source=gradientflow&utm_medium=newsletter) brought it to roughly 90%. It is one specific result, not a benchmark, but the lesson is useful: basic data modeling and documentation mattered more than adding more context.\n\nSome failures that look like AI failures are really old data failures whose consequences have become operational.\n\n#### **Permissions Have to Follow the Data**\n\nAgents also make an old permissions problem harder. A single task may pull information from a warehouse, a search index, a SaaS application, a vector store, or a cache. Those systems do not necessarily enforce access in the same way.\n\nSuppose a source system hides a document from a particular employee, but a search index built from that system does not preserve the same restriction. An agent retrieving from the index can now see something the employee could not have retrieved from the source.\n\nThis is why permissions cannot be treated as something checked only when data first enters the system. When information is copied, indexed, cached, or retrieved somewhere else, [the access rules need to travel with it](https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview?utm_source=gradientflow&utm_medium=newsletter).\n\nThe practical question remains simple: when an agent reads a record, whose authority is it exercising?\n\n#### **Production Requires More Than Retrieval**\n\nSome of the [database architectures I wrote about last year](https://gradientflow.substack.com/p/inside-the-race-to-build-agent-native) make more sense in this context. Databricks is pushing this idea further with an architecture that lets [transactional and analytical workloads operate over a unified storage layer](https://docs.databricks.com/aws/en/oltp/projects/ltap-overview?utm_source=gradientflow&utm_medium=newsletter) instead of constantly copying operational data into a separate analytical system. Less copying can mean less lag between something changing in the business and that change becoming available to an application or agent.\n\nBut I think the broader lesson is more useful than any one architecture. A prototype can show that an agent can perform a task without proving that the surrounding data systems can support it reliably.\n\nProduction has to answer harder questions. Is the state authoritative? How does the agent learn that something changed? Do permissions survive when data is copied or indexed? What happens after a partial failure? Can a retry repeat an action? A working demo can hide how much data infrastructure still has to be built.\n\nFor years, the data stack mostly helped companies observe and understand themselves. Agents increasingly use the same information while doing work.\n\nOnce agents can act on the data they retrieve, the data stack becomes part of the application.\n\n# **More Intelligence, Less Waiting, Less Power**\n\n# **Not Every AI Job Needs a Chatbot**\n\n**[Ben Lorica](https://gradientflow.com/disclosure/)** edits [Ethics.dev](https://ethics.dev/) and the [Gradient Flow newsletter](https://gradientflow.substack.com/), and he hosts the **[Data Exchange podcast](https://thedataexchange.media/)**. He helps organize the **[AI Conference](https://aiconference.com/?utm_source=gradientflow&utm_medium=newsletter)** and the **[Agent Conference](https://agentconference.com/?utm_source=gradientflow&utm_medium=newsletter)**. You can follow him on [Linkedin](https://www.linkedin.com/in/benlorica/), [X](https://x.com/bigdata), [Mastodon](https://indieweb.social/@bigdata), [Reddit](https://www.reddit.com/r/GradientFlow/), [Bluesky](https://bsky.app/profile/gradientflow.com), [YouTube](https://www.youtube.com/c/GradientFlow), or [TikTok](https://www.tiktok.com/@gradientflow). This newsletter is produced by [Gradient Flow](https://gradientflow.com/blog/).", "url": "https://wpnews.pro/news/your-ai-might-be-fine-but-your-data-is-lying-to-it", "canonical_source": "https://gradientflow.substack.com/p/your-ai-might-be-fine-but-your-data", "published_at": "2026-09-22 13:02:33+00:00", "updated_at": "2026-09-22 13:28:35.258621+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "mlops", "ai-tools"], "entities": ["Confluent", "AWS", "dbt", "Gradient Flow"], "alternates": {"html": "https://wpnews.pro/news/your-ai-might-be-fine-but-your-data-is-lying-to-it", "markdown": "https://wpnews.pro/news/your-ai-might-be-fine-but-your-data-is-lying-to-it.md", "text": "https://wpnews.pro/news/your-ai-might-be-fine-but-your-data-is-lying-to-it.txt", "jsonld": "https://wpnews.pro/news/your-ai-might-be-fine-but-your-data-is-lying-to-it.jsonld"}}