How S&P Global Energy used Databricks Genie Agents and FastMCP to turn complex structured data into conversational AI endpoints
• Domain experts curate focused Genie Agents per dataset group without writing agent code, establishing a governed semantic layer.
• Genie Agents act as managed MCP servers that are composed via a FastMCP proxy into composite endpoints for cross-domain queries.
• This architecture significantly shortened time-to-market for conversational data products while preserving Unity Catalog governance.
S&P Global's goal was to fundamentally improve how customers discover and consume insights across our data and research products. While AI powered search and summarization are important, the larger business value comes from enabling faster decision making through natural language access to trusted data, richer cross-commodity analytics, and the ability to connect insights that traditionally exist in separate business lines. AI agents help customers uncover relationships, generate research more efficiently, and derive actionable intelligence from a broader set of information than was previously possible. —Priyanka John, Vice President, S&P Global Energy
If you have ever tried to make a large, complex structured data estate available to AI agents and assistants, you have likely run into the same wall we did: agents are only as good as the context they can reach, and enterprise data rarely lives in one neat, well-documented place. At S&P Global Energy, our data spans Chemicals, Crude Oil, Refined Products, Gas & Power, Liquified Natural Gas (LNG) and more — and each commodity is itself a rich family of datasets. LNG alone includes facility specifications, cargos , outages, supply and demand fundamentals, netbacks, historical and forecast prices, and contracts. Chemicals span capacity, production, utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances. Our other commodities follow similar patterns. This data lives across Databricks and several non-Databricks sources.
Our goal was ambitious but simple to state: make our entire structured data estate available for external consumption by AI agents through the Model Context Protocol (MCP) — so that our customers’ agents and assistants, as well as our own, could ask questions in natural language and get trusted, governed answers.
We evaluated several approaches. What worked best for us, by a wide margin, was Databricks Genie Agents, exposed as managed MCP servers, composed into domain-specific bundles with an MCP proxy layer.
In this post, you’ll learn:
Genie Agents let our domain experts productize their knowledge of the data directly. What used to take a full development cycle now takes days, and every answer stays inside our governance boundary.—Priyanka John, Vice President, S&P Global Energy
Large language models are remarkably good at conversation and reasoning, but they cannot answer questions about your data unless you build a bridge to it. For structured enterprise data, that bridge has historically meant one of the following:
Each of these approaches shares the same problem: the people who understand the data best — our SMEs and analysts — are not the people building the access layer. Every insight had to pass through an engineering backlog. Our time to market for a new conversational data experience was measured in months.
We needed an approach where domain experts could curate and publish conversational access to data directly, engineering could standardize how agents connect, and governance stayed centralized. That is exactly what Genie Agents plus MCP gave us.
Our architecture has three layers, and each layer is owned by the people best suited to it. The diagram above shows the end-to-end flow — including what sits inside the S&P Global Energy network and what sits in the external client environment.
This is where the magic starts, and notably, it requires no code.
Our SMEs begin by selecting the tables relevant to a business domain:
They then group related tables and create one Genie Agent per dataset group — not one giant agent per commodity. Each sub-category of a commodity becomes its own focused Genie Agent. Within LNG, for example:
Every other commodity follows the same pattern with its own sub-categories. Chemicals, for instance, has group-level Genie Agents for capacity, production, capacity utilization, trade, demand by end use and by derivative, inventory change, and country- and region-level supply–demand balances; Crude Oil, Refined Products, and Gas & Power are organized similarly. The result is a fleet of small, sharply scoped Genie Agents rather than a handful of sprawling disconnected AI tools.
Inside each agent, SMEs add the context that makes text-to-SQL actually work in the real world: descriptions of tables and columns, example queries, trusted assets for high-stakes metrics, and business definitions (for example “floating storage is defined as cargoes idling for 3 days or more in vessels travelling below a threshold speed”). This is the step that generic text-to-SQL solutions skip — and it is the step that determines whether users trust the answers.
The key organizational insight: curation became a domain activity, not an engineering activity. The person who knows what “floating storage” means in an LNG context is the person teaching a Genie what it means.
Here is where Databricks did the heavy lifting for us. Each Genie Agent is exposed as a Databricks managed MCP server out of the box, at an endpoint of the form:
https://<workspace-hostname>/api/2.0/mcp/genie/{genie_space_id}
There is nothing to deploy and nothing to host. Each server exposes a small, clean tool surface — essentially two tools per agent:
This two-tool, ask-then-poll pattern turns out to be a great fit for agentic workloads: questions run asynchronously against a SQL warehouse, and the agent polls with the conversation and message ID returned by the query tool until the response is ready.
Just as importantly, these managed servers are governed by Unity Catalog. A Genie Agent — or the user behind it — can only reach the agents and underlying tables they have permission to see. Authentication is handled by the platform. We did not have to build a security layer around our AI access; we inherited the one we already had.
One Genie Agent per dataset group keeps each agent focused and accurate. But real business questions routinely cross groups: “How did the recent outages at Sabine Pass affect cargo premiums into Asia?” touches both the Outages and Cargo Genie Agents at once, and cross-commodity questions like “How are naphtha prices affecting chemical production margins?” reach across Refined Products and Chemicals Genie Agents.
Rather than building one giant agent (which degrades answer quality) or forcing every client to configure a dozen separate servers, we used FastMCP’s proxy and composition capabilities to create composite MCP endpoints — typically one per commodity, mounting that commodity’s group-level Genie MCP servers behind a single server with name-spaced tools. Higher-level composites can bundle several commodities the same way:
from fastmcp import FastMCP
# Each group-level Genie Agent is a managed MCP server on Databricks cargo = FastMCP.as_proxy(genie_mcp_config(“lng_cargo_agent_id”), name=“cargo”) outages = FastMCP.as_proxy(genie_mcp_config(“lng_outages_agent_id”), name=“outages”) netbacks = FastMCP.as_proxy(genie_mcp_config(“lng_netbacks_agent_id”), name=“netbacks”)
# Compose the group Genies into one commodity bundle lng = FastMCP(name=“lng-composite”) lng.mount(cargo, prefix=“cargo”)`` lng.mount(outages, prefix=“outages”)``lng.mount(netbacks, prefix=“netbacks”)
# The same pattern repeats for Chemicals, Crude Oil, Refined Products, Coal …
(Illustrative snippet — adapt to your FastMCP version and auth setup.)
The result: an agent connects to one composite endpoint per commodity and sees a curated set of group tools — cargo_genie_query_agent, outages_genie_query_agent, netbacks_genie_query_agent, and so on, each paired with its genie_poll_response counterpart. The agent’s LLM decides which group Genie to route a question to, or fans a cross-group question out across several, then synthesizes the results.
This gave us the best of both worlds: narrow, high-accuracy group-level Genie Agents underneath, and broad, commodity- and estate-wide conversational access on top.
For our business stakeholders, the technical details above translate into a few very tangible outcomes. Time to market collapsed. Previously, standing up a new conversational data experience meant a full development cycle: requirements, API design, text-to-SQL engineering, testing, deployment. With this architecture, launching a new dataset group — or an entire commodity — means an SME creates and curates the corresponding Genie Agents — the MCP endpoint exists the moment the agent does.
SMEs became publishers, not requesters. The domain experts who understand LNG cargoes or chemicals supply–demand balances no longer file tickets to get their data exposed; they curate a Genie Agent and it is live. Engineering effort shifted from building bespoke access layers to maintaining one thin, reusable proxy layer.
Governance came built-in. Every question an agent asks runs through Unity Catalog permissions, on governed tables (native or federated), with full auditability. Making data available to AI did not mean making it available outside our controls.
One integration pattern, many consumers — inside and outside the company. Because MCP is an open standard, the same composite endpoints serve our internal agents, our customer-facing AI experiences, and — critically — our external customers, who can connect their own MCP-compatible agents and assistants directly to governed S&P Global Energy data. We built the bridge once; every MCP client, internal or external, can cross it.
Answer quality is measurable and stays that way. In a domain where pricing, supply, and contract data drive real decisions, users need to trust every response and calculation. Genie Agent Benchmarks give our SMEs a built-in way to define test questions that mirror how users actually ask questions (including multiple phrasings of the same question) and score the agent's accuracy automatically against verified answers. Just as important, benchmarks can be rerun after any refinement to instructions, data, or business logic, so accuracy is maintained over time. The result is an efficient continuous quality loop: curate, benchmark, improve, and re-benchmark to ensure our agents stay accurate as our data and our customers' questions evolve.
A few practical takeaways from our journey, for teams considering a similar path:
Our goal was to make our entire structured data estate — spanning LNG, Chemicals, Crude Oil, Refined Products, Gas & Power, and more — available to AI agents: securely, accurately, and fast. With Databricks Genie Agents as the SME-curated semantic layer, managed MCP servers as the zero-deployment integration contract, and a FastMCP proxy for cross-domain composition, we achieved exactly that:
The deeper shift, though, is organizational: the people who understand the data are now the people who publish access to it. That, more than any single technology, is what turned our structured data from something users query into something they can simply talk to.
Subscribe to our blog and get the latest posts delivered to your inbox.