The Reconciliation Problem Is How Agentic Analytics Platforms Grow Up

A practitioner who rebuilt an agentic analytics platform four times concluded that reconciliation and governance, not data plumbing, are the core obstacles to making multi-source AI analytics reliable. The first version, a single MCP server placed in front of a semantic layer, failed in three ways: it inherited one source's blind spots, exposed whatever the service account could see including customer emails, and returned stale and fresh answers in identically confident prose. The fix was to make every MCP server its own enforcement point, with each call carrying the requesting user's role and scope as a first-class parameter checked before the underlying source is touched.

Note: specific figures and incident details below have been generalized to protect proprietary information. The architectural patterns, design decisions, and technical tradeoffs described reflect real implementation work. The Slack thread that started this was about a gap between two revenue numbers that shouldn’t have existed. The warehouse and the BI dashboard were reporting different weekly revenue growth for the same period — not wildly different, just different enough that someone was going to ask why. Neither system was wrong, exactly — they were quietly answering slightly different questions — but I was the one who had to explain that in a thread with a VP in it, which is not a fun way to lose an hour. That kind of gap shows up everywhere once a company has more than one system that can answer the same question, which by a certain size is every company. An analyst ends up doing what I’ve come to think of as the tab dance: warehouse in one tab, BI tool in another, sometimes a partner report in a third, copying numbers into a scratchpad and eyeballing whether the difference looks like normal drift or an actual problem. Nobody’s cutting corners. They just don’t have a system that does the cross-checking for them. I decided to build one. It took four rebuilds before it actually held up, and each rebuild started because the previous version had broken in a way I hadn’t planned for. The first version was almost embarrassingly simple. I stood up one MCP server in front of our semantic layer, pointed an agent at it, and let people ask questions in plain English instead of writing SQL or clicking through a dashboard. It was a good demo. Everyone loves a good demo. It broke in three specific ways once real people used it for real work. First, it only had one source, so it inherited that source’s blind spots: if the semantic layer’s revenue figure was noticeably stale, the agent stated it with exactly the same confidence it would state a fresh number. Second, it could see whatever the service account behind it could see, which was, generously, everything. The same MCP that answered “what was last quarter’s revenue” could also be asked to dump a table with customer emails in it, because nothing in the plumbing distinguished those two kinds of question. Third, and this one took me a minute to even notice: it never told anyone it was guessing. A stale answer and a fresh answer came back in identical, confident prose. That third one is the one that changed how I thought about the whole project. This wasn’t a chatbot problem. It was a governance problem that happened to be wearing a chat interface. The obvious fix for “only one source” is more sources. I built MCP servers for the other places numbers actually live: the raw warehouse, the BI tool’s cached metrics layer, a couple of internal APIs for product usage data. Now the agent had real reach. It could go get the numbers a human analyst would have gone and gotten. It also multiplied the exposure problem by however many MCPs I’d just added. Access control had been living entirely in the BI tool’s UI — row-level security on a dashboard, folder permissions, that sort of thing. None of that travels with a raw tool call. I remember the kind of ticket that made this concrete: someone asked the agent a simple, account-level question and got back fields well beyond what the question needed. Nothing malicious happened. The MCP calling the warehouse directly just had no idea it wasn’t supposed to hand that over, because nobody had told it. The plumbing is never the hard part. The governance is, and it only becomes visible once the plumbing actually works. The fix was to stop treating access control as something that lives upstream in a BI tool and start treating every MCP server as its own enforcement point. Each call now carries the requesting user’s role and scope as a first-class parameter. The MCP checks it before touching the underlying source, the way an API gateway would, instead of trusting that whatever called it already checked. PII gets handled a layer earlier than most people expect. Every MCP response passes through a redaction step before it lands in the agent’s context window at all: entity detection on the way out, not a prompt asking the model nicely not to repeat sensitive fields. A field either clears the filter or the model never sees it. I don’t rely on instructing the agent to behave. I rely on it literally not having the option. Here’s the part that actually surprised me. Even with clean permissions and clean redaction, the warehouse MCP and the BI-tool MCP still disagreed on plain questions like “what was revenue last week.” Not because either one was broken. Because “revenue” meant two subtly different things to two subtly different systems — one included a category of adjustments the other excluded, and the two refreshed on noticeably different schedules. A number pulled early in the day could legitimately be most of a workday out of sync with the source it was supposedly mirroring, and neither MCP had any way of knowing that about itself. A human analyst absorbs that kind of drift by instinct, built up over months of learning which number to trust when. An agent doesn’t have the instinct. It just picks one and states it with total confidence, which is worse than a human’s uncertainty, not better. The fix was an actual semantic and ontology layer, exposed as its own MCP. Not a wiki page describing what “active customer” means, but an enforced contract every other MCP has to resolve its fields against before returning anything. If two sources can’t agree on the definition of a metric, that’s not something the agent gets to paper over with confident prose — it’s something the platform has to resolve once, upstream, the same way you’d settle it for two humans arguing in a meeting, by pointing at whichever definition is actually the source of truth. Once governance and shared definitions were in place, the speed unlock became obvious. The agent doesn’t need to ask one source, wait, ask another, wait, and let a human eyeball the gap. It fires the warehouse MCP, the BI-tool MCP, and where relevant an external benchmark, in parallel, resolves every field against the same ontology, and runs an explicit reconciliation step. Not “pick whichever number sounds right” — compare these under a known tolerance band, and if they disagree past that band, say so instead of quietly choosing one. We settled on a tight tolerance band for revenue-class metrics, and something looser for anything derived from sampled data — thresholds a human argued about in a review meeting, not something the agent invented on its own. That’s the actual replacement for the tab dance. Not a faster way to ask one dashboard a question, but a system that does the cross-checking a careful analyst would have done, in seconds, and shows its work when the sources don’t line up instead of quietly hiding it. The last MCP I added was web search, for the cases internal data genuinely can’t answer on its own: a competitor’s pricing move, a macro trend, a benchmark nobody inside the company tracks. This is useful and a little dangerous in the same breath, because an external estimate dressed up in the same confident prose as an internally reconciled number is exactly how you end up with a board deck citing a guess as fact. So estimates get their own confidence tier, not a footnote. Anything sourced externally is labeled as an estimate directly in the response, kept separate from the reconciled internal number, and never allowed to quietly raise or lower confidence on a figure that came from governed internal sources. Close is genuinely useful for a lot of decisions. It just has to say it’s close. By the third or fourth recurring investigation — a revenue variance check, a churn benchmark against market data, a pricing anomaly review — I noticed I was rebuilding the same shape by hand every time: which MCPs to call, which RBAC scope applies, how to reconcile, how to label the external piece. That repetition was the actual signal that something should become a named, reusable Skill instead of a prompt I re-explained from scratch each time. A Skill, in this platform, isn’t a prompt trick. It’s a bundle: None of this is finished. The Skill catalog is still a handful of patterns deep, memory versioning across investigations is still half-built, and I fully expect the ontology layer to need another pass the first time someone asks a question that doesn’t fit its current shape. What shipped isn’t a faster chatbot in front of a dashboard — it’s a platform where every MCP enforces its own access control and PII boundary, where a semantic and ontology layer is the referee every source has to answer to, where reconciliation across sources replaces manual cross-checking, and where the recurring shapes of that work get packaged into Skills instead of re-explained by hand each time. The gap that started all this would still show up today. The difference is that now the agent would tell you both numbers, tell you why they disagree, and tell you which one to trust, before anyone had to open a second tab. The Reconciliation Problem Is How Agentic Analytics Platforms Grow Up https://pub.towardsai.net/the-reconciliation-problem-is-how-agentic-analytics-platforms-grow-up-d575c1989b61 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.