Presentation: Autonomous Data Products for the Autonomous Era: Rethinking Data Architecture for GenAI

Jorg Schad, speaking at a presentation on autonomous data products for GenAI, identified four key challenges in moving AI prototypes to production: standardization of data access, managing the growing ecosystem of models and protocols, avoiding context rot by adding only relevant data, and ensuring safety to prevent data exposure or system damage. Schad argued that most GenAI projects fail not because of model quality but due to underestimating data operations and access complexity.

Transcript Jorg Schad: We're going to be talking about actually two of my favorite topics, and that is data meets AI. We'll be talking a little bit about, what do we need? What do we need to actually make data accessible to GenAI? In my experience, most reasons why GenAI or even before whenever I feel we say GenAI nowadays, the same was true to deep learning, to analytics before. Why most of those projects fail, it's not because of I have a bad model. It's not because I'm choosing the wrong thing. That might happen at times. The most consistent failure mode I've seen is because we underestimate the economy around it or the operations around it. I think the probably most important piece to that is, how do I get access to data? I've seen a lot of prototypes. People prepared it either internally in the company or even external consultants. They come in, have this really cool presentation. Once we really want to turn that into production code and want to connect it to our real data, not just this little cleaned up sample data we had before, this is when actually things break apart. This is, I think, the only switch to GenAI where I would see the difference is actually like the speed at which that's happening. All of a sudden, it's not just like a data scientist anymore who goes in, who might even have a look on the data. It is actually autonomous agents, ideally, consuming my data, connecting to my agents. Just like this speed of data access becomes much faster. Therefore, also the scale. Ideally, our dream, our CTO's dream is usually like we have a fleet of agents accessing our dataset. That, of course, is also adding to the complexity. The Bumpy Road from Prototype to Prod How does it often look like, that bumpy road from prototype? I've often seen and even heard recently, there's like a new project, large corporation. You have some consultants coming in. They're building this really cool notebook. They show a lot of value. They present it to upper management. Upper management is really excited. You actually fail to turn that over, because all of a sudden, you have to connect to other data sources. You have to connect to other systems, and it's not working as a small, nice environment. The other aspect is just that field is growing, both in terms of ML models, ML applications, systems to connect to, different MCP, A2A, so different protocols and framework. Just the ecosystem is developing really fast. As we said already, also in terms of like the different data sources we might want to onboard. If we are in an enterprise setup, I have one data source, but then all of a sudden, it's like, how can we make another data source available to that? How can we connect to another data asset here? Just the sheer volume of things to connect to. Sheer volume is actually the next problem. Once we have connected all those things, it might actually be too much. We're talking a little bit about that. Just the context rot. The more we actually add to LLMs, GenAI, and to a degree that's also true to deep learning and other systems, the more unspecific context or data we add, the lower the performance will be. There's a threshold. We want to add the right amount of information, that's critical, but also not too much. Then last, but probably most important is the safety aspect. How can we do that in a safe way where not our coding agent is wiping out our databases? Or maybe even more impactful, we expose a lot like private or sensitive data to a chatbot or to people who shouldn't have access to that. If I would name those four pillar problems, it would be standardization. How can we standardize the access to data, to ML systems? I've seen a lot of enterprises where basically each team, they're starting to build their own stack. They're choosing their own tools. One is choosing LangChain. The other is writing their own things. They have different interfaces, different interfaces to data. How can we actually standardize that to scale? Next is just the speed of access and speed of being able to connect to different tools. Again, this is also related to standardization, of course. Next is the specificity. How can we expose specific information? Just the right amount of information, but not too much. Then last, is the safety aspect. How can we make that in a safe way? I think this is again, where I think there's a big difference to when I have a human looking over it. As a data scientist, they might actually realize, we are exposing data here. As soon as we are on this autonomous path where agents are autonomously dealing with data, safety becomes a really critical aspect. Background I'm Jörg. I'm head of engineering, VP engineering at Nextdata. My background is building large-scale distributed systems, in particular, database systems. I said I'd been building early on Apache Mesos, early on Kubernetes. I think by now all my code is out of Kubernetes. This feels a lot like what we are dealing with right now. I think for me, Docker, Kubernetes, they were the advent of standardization and also standardized specifications where different people could implement against in the microservice world. I feel nowadays we need something similar in the data world. Containerization, so how can we contain data assets? Also, how can we specify them in a scalable fashion? Then the second aspect, which Kubernetes brought into the game of microservices or also Apache Mesos back in the days, but I think nowadays it's synonymous with Kubernetes is the orchestration aspect. How can I orchestrate those different containers? How can I manage them in a global scale? This is what we'll be talking about today. The Data 2.0 Problem Let us recap a little bit that problem of like data 2.0, so basically, our current data architectures, why do they fail? If we first start taking a look from an organizational level, like if we want to implement a new ML, GenAI initiative, who is all involved? We have platform engineers. We got the data product owners. We got the ingestion team. We got data architects. We got data stewards, data engineering, data governance, consumers, and then in the end also just the data consumers as for example, agents. All of them actually face their own challenges because we have a very fragmented system, and all of them are dealing with one slice of this big problem, and they all need to coordinate amongst each other. We already see that the data and metadata is split in between those different personas. That actually is really slowing down the entire process. If we take a look at the same problem but just from the architecture perspective, we end up with like this data management hairball. We have a lot of different tools for cataloging, for data governance, for data quality. We might show in like Monte Carlo, Soda, Great Expectations. We have different qualities. We put on another solution for data lineage. We have either one or potentially multiple solutions for data pipeline and ETL. Now our team, they all have to weave that together into this one big hairball, into this one big architecture. As I said, it's like the other perspective on this, whereas this is a bit more focused on the organizational side of things. Who is actually involved with this? This is like the resulting architecture view on that from more like a technical side. Luckily, I think we already have a number of tools out there. We have a pretty good infrastructure, like multi-cloud infrastructure, I think is fairly well understood. We also got data foundations already. Most enterprises, we already have storage. We already have compute. We already have security down there. We have figured that out, but how do we all turn it together? Then, how do we actually make that accessible to a growing number of different applications on top? It always reminds me of the LLVM stack, the compiler stack. Anyone familiar with that? Like the C compilers. You have a number of different frontends on top. You can compile to different targets. You have a number of different source languages underneath. Then they have that intermediate representation where they can do all optimization. We basically end up with this hourglass of standardization. I think this is very similar of what I think we're envisioning or also what I would hope for. How can we put this middle layer, how can we make that into a standardized layer and then we can build different things on top? Again, just taking the Kubernetes side here, with the Kubernetes view. What did Kubernetes and what did Docker do back in those days? They actually went in and they standardized container images, container runtimes, network layer, storage layer, and so also different people could implement on it. We can have multiple tools underneath and we can also have different frontends on top consuming from it by standardization. That kind of complexity, I think we have very different and many aspects contributing to that. Our data complexity is growing based on the number of different dimensions and different options we have in those dimensions. From locations, where is my data stored? How many cloud regions do I have? How much of my data needs to, for example, stay in our own data centers? Then the formats and different access modes of data. How can I access my data? Through RAG, through MCP, vector embeddings, SQL Notebooks, Power BI. Processing, so how can I actually compute my data? Do I have Spark? Do I have Polars? Do I have a large Snowflake database somewhere for warehouse processing? Am I building on top of an Iceberg Lakehouse? Where do I actually process my data? We have different applications on top. Then, also, I think we should never forget the human side of that. We also have different human skills involved and different personas. If you remember like our first organizational slide actually interacting with that platform. That's like that complexity. This might seem like this complexity is only relevant for us as humans or for us building such platforms, but, in the end, it is relevant for agents or this autonomous consumption. Context Rot: How Increasing Input Tokens Impacts LLM Performance We talked a little bit about this context rot, and this is, I think, the natural way which I see happening a lot. Like, we exposed our data warehouse via one MCP endpoint. We have so many different MCP tools out there. If we onboard someone new to our company, would we give them the entire company handbook, or would we give them specific information they need for their onboarding? I think we need to think similarly about LLMs or agents in particular, of what information do we make accessible to them? These are some LangChain benchmarks where they just kept exposing more and more agents, or in more and more context to those agents. We can see like the performance is actually decreasing. Typically, there's like a sweet spot of how much information the LLM can deal with. It comes down to like, if I have access to all the specific information for a particular task, this is my sweet spot I want to be. If I expose more information, it becomes hard to choose. We end up with hallucinations because the LLM can't decide what is relevant here or what not. How can we expose from this complexity here? How can we break down that complexity that it actually becomes feasible for autonomous consumption to deal with that? Safety, already talked about it a little bit. How can we make sure that in this autonomous world, it's the same for like human consumption. It just becomes much more important, much more critical if we talk about autonomous consumption. How can we make sure that the data access is safe? We still enforce data quality rules. We still enforce data privacy rules across this really fast autonomous access of our data. Taming Complexity in Computing, now in Data How have we solved that before? Before, in like taming complexity in computing, I think a natural approach, also if you look back at history, was always like, we try to encapsulate it into manageable units. We try to abstract it by clean interfaces. Then we try to automate it. That for me was like a lot this journey towards containerization. We have actually written containerizers, as they were called in Apache Mesos, before Docker was super popular. I remember actually writing those things before Docker was super popular. They were around already. What does it help us? We can encapsulate it into manageable units. We actually know what we talk about and we have one well-defined interface to all aspects of it. I think in particular, if we talk about data, what do we mean by if we say data? We can mean the raw data, like the actual business data. We can talk about metadata, like when was it last accessed? What is the data classification rules to that? It's typically not just the data we want to have access to, but also certain metadata around it. Abstraction, how can we define that by clean interfaces that I don't have to worry about how to access that? In that big stack, or also in current enterprise architectures, I have to know, where is my data stored? Which team do I have to talk to? Do I have to get a Snowflake access? Do I have to get ADLS access in order to access the dataset? Same again with metadata. How can we actually abstract that via a clean interface? Then, once we have that, I think the automation part becomes a lot easier. How can we move typical operations away from human people and can build processes around that, runbooks around data for classifying data? Which data should be exposed through MCP or whatever protocol comes after, to autonomous agents? What data is ok to be put into vector embeddings and accessed by our chatbots? Once we have all of this, we can actually start automating it. Autonomous Data Products This is the main idea of autonomous data products. How many of you have read the "Data Mesh" book by Zhamak? Data products, they're like one of those central units combining both data and metadata. The idea for data products in general is, I want to have a single asset for data. Like if you ask me, how do I get access to this particular data asset? I can just give you a reference to that data product and you don't have to worry about where the data is actually stored, but you know how to get access to that. A data product is the equivalent to a container in the microservice world. It encapsulates data, metadata, access paths, and it gives us that abstraction layer we talked about earlier. If we look at what an autonomous data product is, if you start Googling data product now, I think there are so many different definitions. Some people just define, if I have a dashboard somewhere, is that a data product? Some people define a table being a data product. The term autonomous data products was our way of encapsulating what was meant in that book, and differentiate from other terms of data product in general. What is the idea or definition of an autonomous data product? I think, most importantly, it is a running process. Each data product has a runtime controller or what we also call kernel. It's like a little operating system kernel, like backing and managing the lifecycle of such data product. The lifecycle of a data product, it basically starts by sensing inputs. It can sense upstream changes in the data, like a watcher. If it's depending on another data product, it can sense changes in that data product. Then I can start configuring, when do I want to update? When should this actually run? We'll see that in a more concrete example in a bit, but this is like the first stage of the lifecycle. Next stage of the lifecycle is in actually running the transform. This is happening down here. The data product is just a logical abstraction on top. It's not replacing the Spark job. It encapsulates a Spark job, for example, or a Databricks transformation written in SQL. Basically, then after it sensed input and it decided it needs to run, it would actually trigger a transformation down in the underlying compute layer. Very important is the outputs of that are not immediately accessible to an end user. There might've been failures. There might be like data quality checks afterwards. This is in the next lifecycle phase of a data product. It will actually start checking, are my data quality rules in sync? Are they actually valid? Then once this is true, promote, as we call it, the output, making it accessible to downstream or also external consumers. Why is that important? In this current world, what's happening is my Soda, Monte Carlo, Great Expectations, they might cause an alert. This is usually post-fact. Like nowadays, typical data quality tools, if they tell me your data is wrong, it is already accessible by downstream consumers. My chatbot, my agent might've already taken certain actions based on it or answered questions based on it, and we have to fix it post-fact. This is one of the ideas of this lifecycle that is actually happening as part of the lifecycle before data gets accessible, consumable by anyone external of the data product. Data product also is exposing standard APIs for observability and discovery, so I can identify what is in a data product? What is the metadata around it? What is the code behind that transformation? I don't have to jump between a number of different systems to actually find that out. This is the encapsulation part we talked about. Then, last but not least, talking about the downstream consumers, different downstream consumers often require data in different formats. If I'm a data scientist, I might prefer dealing with my data through a SQL interface. Or, if I'm a data analyst, I might prefer the SQL interface somewhere, Snowflake or more warehouse style. If I'm writing a new machine learning model, then probably some file-based access is great. If I'm an autonomous consumer, I probably prefer some embedding output or even something exposed through MCP, or if there's another protocol down the road, that other protocol down the road. Different consumers might require different modes of data access. This is the other aspect, this logical abstraction a data product gives us, because it's centered around the semantic representation of the data. If I'm discovering a data product, I care about what data asset is available. In second steps, then I care about how can I consume it, that distinction between the logical representation and the physical representation. This is actually available in different formats for different consumers downstream. We've already said it's all API driven. Each data product itself is offering its own API. If I want to share a data product, I can just hand over the URL to someone. If that person has access, he can just work with that data product in particular. It's all API driven. All of this I can do, get the information, get the different models, get inputs, outputs, the current status, and also the events tree. I can do that through a very simple OpenAPI based endpoint. Data 3.0 How does it help us abstract away this hairball in the middle? It gives us just a standardized way and an encapsulated way for autonomous data. It allows us to actually be domain oriented. There's a set of multiple data products. Each domain team coming from domain-driven design can focus on its domain of data products to offer its own expertise to anyone else. It removes this bottleneck of a central team. When I want to publish something, the central team is actually responsible for. It shifts left that responsibility to the actual business teams who know what is the semantic of the data, who actually can deal with that. We've seen the different slices of the different organizational pieces here. How does it actually come together? We have the data product developer. They're the people actually building a data product. They build and run a data product. We have the data product owner. They're the persons actually responsible either for that business domain or the general mesh. Often, if it's a smaller mesh, it's just one person for the entire mesh. They actually go in, they will model and manage. They will check for compliance. They will check how often was this used. They have a single plane of view to get that. They don't have to jump through 10 different systems to see how often was a particular data asset being used. Then we have the data product consumers, either agents or humans, and they can discover, they can use, they can request access through a common interface independent of the platform underneath. Then we have the infrastructure team. Nowadays, the infrastructure team, they're often tasked with actually giving access to the dataset. They have to debug what is wrong in a particular statement. They are often like that first point of contact there. With the data products, they only need to provide a self-serve infrastructure for data products to be built on top. There's an abstraction of like an infra service. For example, if I have a particular Spark instance, I make that available to the data product developers through an infra profile. As a data product developer, I don't have to worry about where is that running, how is it defined. Basically, I use an abstract infrastructure service. Similarly, vice versa, the infrastructure team, they can really focus on providing the infrastructure, and they don't have to worry about what is really written on top because they don't have to debug it in the end. Example Let us look at an example in context. Here we have a data product for customer feedback. It's using two input data products, one being very much focused on structured feedback. For example, imagine Amazon reviews coming in almost like on a minute basis. This is stored in a very relational format because it's a standardized form. On the other hand, we've got more unstructured data. Imagine we also have like point of sale, actual physical stores, and people can also provide feedback there on a written form, and then once a week, we actually get scanned PDFs back. Our customer feedback data product is actually using those two data products as input. As you can already see, first of all, it's very different data modalities. It's also different speed of data arrival. That already leads to that question when we talked about sensing input. When do we actually want to update whatever we expose here? Do we want to do that in real time or whenever we get an update here from a new Amazon review? Maybe we want to do that. Maybe this is a bit too fast, because it actually doesn't give us that much new information. This is really dependent on the use case. On the other hand, do we just want to update whenever, like once a week, every Friday, we get those PDFs over from all the stores? Maybe this is a bit too slow in the end. By actually being able to sense and then devise a policy when to run on this data product level, it actually allows us to configure that. I've seen so many Spark jobs, large-scale Spark jobs, which just run plain every hour. This is great for Databricks because they get a lot of money through that. It's not great for the enterprise because we often waste a lot of compute resources as the output is only accessed maybe like once a month when the CEO actually wants his or her report. I think this is a very powerful thing to actually have like run policies when we want to spend time on compute or transformation. Then, we basically run that, we aggregate that information. Before we make it accessible to the downstream consumers, we are running our data quality checks. These are basically data contracts. I can also leverage systems. There's a standard interface in there for supporting Monte Carlo, supporting Soda, Great Expectations. Those things are run before data is being made accessible here in the consumption layer to any downstream consumers. This is maybe the last part here. As we have different consumers, we actually want to make it available in different formats. We have our embedding output of the same semantic data model. Of course, it is not looking like the same because we've got a bunch of numbers there. In the end, it's representing the same customer feedback dataset. We might have a SQL output for our data scientist. We have a document file representation for model fine-tuning and training. Then we have an MCP output for also autonomous querying of this dataset. Then, anyone can build LLM-based agents against it. Safe Autonomous Data Consumption It sounds great so far, but how can we make this accessible in a safe fashion? Our data product developer is building that, is very proud, but now kind of like CISO or someone from the governance team is going to come and is like, no PII can be processed by the LLM. We end up in this challenge of, we want to enable data products developers, the domain teams, the actual business teams, to develop something as quickly as they can and be as independent as they can. On the other hand, the governance team needs to ensure that our data consumption stays safe and we are not exposing data anywhere outside. How can we bridge that? In current implementations, this is often being done. This guy is developing this. Someone from the governance team is reviewing it, giving thumbs up, and then it can be released. This is drastically slowing down the release cycle of those things. Because the governance team typically becomes a bottleneck in reviewing those. The idea behind data mesh or behind autonomous data products is that I can independently define so-called policies. Policy is defined upfront, and that can be encapsulating, for example, no PII can be exposed to LLM. Then we can have a central contract repository. Each data product which wants to be deployed on the mesh needs to adhere or needs to implement those contracts. For example, there's no PII contract. In pseudocode, you could imagine it says, if you have either an MCP output or if you have an embedding output port, then you need to have our predefined PII check implemented on your data product. This is actually checked whenever you deploy something to the mesh. It's not that the governance team needs to review all of that. It's like an invariant to every data product running on the cluster, because those policies can be centrally defined and then are enforced by the orchestration system. If you want to imagine, if you're familiar with Kubernetes, Admission Controller in Kubernetes, which actually checks something before it's deployed on the cluster. The only slight difference is this is actually in runtime check. If you're adding new policies during runtime, it will also still get exposed or checked during runtime. I think this is like the way to keep things safe but scalable and fast at the same time. Because otherwise, your governance team is going to become the bottleneck out there. Agent Lifecycle with Progressive Tool Discovery Also covered the safety aspect. Now let us see what is the implication of that for our agent lifecycle. We talked about this problem of specific data being made available. Right now, we just said we make it really easy for people to build data products with MCP output ports. I think the first important aspect of this pattern is we have standardized how we build MCP tools and output ports. Because they are being defined as part of the data product, there is this standardized way of actually then building or exploiting the tools. We separate what we want to achieve from how. I as a data product developer, I can focus on, I want to expose these MCP endpoints. Then the central team or the driver, as we call it, can decide how is that supposed to run. I think this is the one part. We, A, standardize it and make it easy for people to build it at the risk of having a lot of different MCP tools being available. How can we deal with that? This is then the lifecycle with progressive tool discovery. How many of you have heard about progressive tool discovery? In short, basically the MCP protocol. This is the same for other protocols. If I say MCP, I personally believe there need to be multiple iterations on that protocol to become really useful. Whenever I say MCP right now, this is pretty much also applicable to other protocols. That has the standard way to interact with that. It has an endpoint called list tools. When you talk to an MCP server, you can call list tools. This is actually going to tell you all the tools you have available. The nice thing about this list endpoint, you can also subscribe to it. It doesn't need to be a static list of tools, but it can change over time. That actually enables you to implement a pattern called progressive tool discovery. How does it look like? First of all, we have our entire mesh. That mesh actually has a single endpoint for all data products and all tools being exposed, an MCP gateway. If we connect to that, there's only a single call available. You can imagine this is called discover tools. Our first call to this list tools endpoint is just going to return a single tool, and it's called discover tools give me context. What that basically says is if we then call that with a specific task we want to achieve, we recall that endpoint, and then next, as we are subscribed to our list tools endpoint, we actually get more tools, but these tools are very specific to our task. In this case, it's for example, like get customer feedback. The rest is a bit prefacing base to the system. We now have an endpoint available, because we've said we want to get customer feedback. We can now actually have another tool which we can use for that. The LLM can then choose to actually leverage that. It's going to go through the background. It's going to get some data. The nice thing is the access is all controlled through the single gateway. Again, we're in standardization. Not each team is implementing their own way of authorizing, standardizing the access. Through that gateway, there's a single way of enforcing who has access to what, and what can get exposed. We get the data. We can make our decisions. Then we basically start over again. After we are done with that loop, we actually reset or re-filter the number of tools which are being made available. This enables to only make specific tools or resources, whenever I say tools right now, it's actually tools and resources being made available to the agent, to the autonomous consumer. There we really avoid overloading that autonomous consumer. I think we covered all kinds of basic building blocks. I think it's, first of all, encapsulating data, standardizing access to data, making multimodal access available, but also still standardize and filter who can access what data. I think those are our building blocks for Data 3.0. Data 3.0 Autonomous Abstraction Then, what does that enable us to do? What is kind of like turning that into business outcomes, or business processes? In our current data stack, everything is centralized, storage-oriented. The first decision we take is, do we use Snowflake or do we use Databricks? Data 3.0 enables you to really go from like a semantic domain-oriented model first. What data do we want to make available? Then, second decision is, how do we want to make it available? What is the physical representation of that? By the encapsulation we talked about, because we have data available together with the metadata, we actually get context-aware ground truths which scales for our access. We're not exposing our entire data warehouse, our entire, like all the tables within Snowflake, where we really expose only very small, specific slices to the autonomous consumer. Modality shift. I said like in the Data 2.0 stack, we built a particular application, often like even a particular storage technology for one particular use case, because we were focused on, we implement that as a Snowflake table. By being the semantic first model, we built a data product. We actually enable like reuse in a multimodal setup. If we now need it as an embedding output, we are in a stage, we have some SQL table being generated, and now a new business requirement comes and says, we also need that as a vector embedding. What do we do? We are not building another pipeline parallel to that. We are actually just extending the data product with another output port, and make a new output modality available to that. How many of you have suffered through that, if you had like multiple representations that actually like, one representation was out of sync with the other? I've seen that fairly often, because we have two different pipelines. One pipeline is generating the embedding outputs. The other is generating our SQL table or filling our SQL table. One pipeline breaks, one pipeline runs slower, and all of a sudden, our SQL is on the version of today, whereas our embedding representation is still two days old. As soon as we have systems that actually consume from both and realize that they are in sync, this will fall apart. That's another advantage of the encapsulation because we encapsulate all of that in one. All those outputs are always in sync. If we update one output, we update all of them in lockstep. I think that's that. Orchestration shift. I think this is, again, what we just talked about. Instead of building separate pipelines, one for embedding, one for SQL, we actually have the self-orchestration aspect. Like each data product is sensing its input. Each data product can decide when it wants to run. We are turning this power, this decision when to run from that central team, which would build those pipelines, to the data product owner. This is, again, the shift left where we want to empower developers who are close to the business domain to decide when they want to run. Then from the reasoning shift, or for me, this is a little bit that architectural shift as well. Because before in the old world, we had to build on lineage on top. Why did we have to do that? Because we had so many different systems coming together, so we needed another solution, just to track lineage across those different systems. Because here we are focused on the data product abstraction. We have lineage inbuilt. It basically comes for free through the system. We don't have to build something on top. It is in there. That also allows us, for example, for data quality or just also reasoning. If I have to justify why was a certain decision made, I can always go back and say, this was because of data version X, model Y trained on this data version. I basically have lineage throughout my entire data history for a particular decision being taken by a model, for example. Then, last but not least, the governance or also data quality aspects. Remember that example where our team is getting a page because Monte Carlo, Soda, and Great Expectations, they detected something as being wrong. At that point, the data is already consumed. It's after the fact. Again, this is not against any of those tools. It's just by the way we actually use it. By using the encapsulation of data products, it's happening within that process of a data product, within that lifecycle before it can actually get consumed by anyone. Because we want to enable all those agents who actually don't really understand what they're consuming. They're just doing it. A human might actually still recognize, this looks really wrong, ideally. An agent's very unlikely to detect, this data looks wrong. This is why I think this central governance is really crucial for the autonomous consumption by agents, LLMs, chatbots. Demo This was like a lot of foundations. I wanted to turn it a little bit into reality. This is now a particular system. This is what we are building with Nextdata. I think it's important to keep in mind, this is one implementation of it, similar as Kubernetes is one implementation. Apache Mesos was another implementation for that orchestration principle. OpenShift nowadays is another one. This is just one particular implementation of that pattern. I strongly believe that the underlying pattern, they actually scale beyond that. First thing we see when logging in, we see the different business domains. Each of those bubbles is representing a business domain. Let me zoom in so we can actually see something. For example, up here we have our retail domain. That retail domain can still have multiple subdomains. For example, here in the sales subdomain, we see the individual data products. If I zoom out a bit, we see the different models they are offering. We can already see the lineage lines. These lineage lines, this is at a very high granularity. This is basically data product lineage. Do we have an example where we actually consume from? Here we have an API endpoint. Also, for external data sources, we get a quick glimpse of, where is my data coming from? How does it interact with other datasets? Then if we look at one particular data product, so let's, for example, take this one. The first thing we see, it's just like a quick description. Right now, I would wear my hat of, I'm a data consumer. I want to figure out, should I be using this dataset? First of all, I can go in, I can try to understand, what is it doing? I see that it started. I see that it's fairly recently updated. I can see the owner whom I could also contact if I have other questions. I also see all the different storage and compute services related to it. If I wanted to, I can even look at the code. I'm not logged in. This basically allows me, if I'm technical enough, I can actually look at the definition of that data product. Next thing I will see is a trust summary. How often has it been accessed? Also, how often is it passing its own checks? Does it actually live up to the data quality it promises? Then, after I've gotten that first sense of trust, the thing which most other things are centered around are actually the models, the data models it's offering. Here, we only have one data model, but I can see it's like new documents, and I can see what it is. I could now also go into more details about that. We can maybe jump to another data product later to see some references. I can also see, for example, document or product ID, whether those are centrally defined topics out there. Related to that, I can also see the API functions. API functions are like, what are the APIs I can directly interact with this data product? In this case, this will only be implemented against MCP right now, but these are like the questions as a data product can answer best. Instead of someone else building an MCP server later next to it, I as a data product developer, I can say, these are the questions my data product can answer really well. It is actually stored together with that implementation. Then, next, what I would probably do if I want to assess a data product, I've now decided, yes, this is the semantic data I actually need. Next question I have, how can I consume it? These are the different output ports. In this case, I have it on Snowflake, I have it on Iceberg NS3, and I have it as a vector embedding in Pinecone. I could now also go in and actually request access to that if I have the right permissions. I could gain access. There's a plug-in implementation for that, and the process really depends on different end users. It might raise a Jira ticket and someone has to approve that Jira ticket, or it can be an automated process dealing with that. I can also see, how can I consume those API functions? In this case, it's being made available through the MCP API output port here. Next, I would actually look at, what is this data product promising me? What can I rely on when I use that data product? These are the data contracts associated with that. I can also see where we have this information available, whether they have recently passed on the last run. Also, these ones are green. They're just the external checks, so they look slightly different. Here we're using Soda, for example. I can basically see, does that data product live up to its promises? Last, I can also see which of those global policies are being satisfied by this data product. Here, for example, there's a central check saying data completeness, and also the compliance output that we're not exposing like email or social security numbers. This data product is implementing all those checks, so it's passing those policy checks. Our data governance team, our CISO can sleep well at night dealing with that. I think that is a quick overview. How does it connect to LLMs? Here I have my Claude Desktop connected to it. I can basically ask questions, what data products are available in the demo mesh? While this is computing, let me already type the next question. What data products have been accessed most in the last week? I think this is an important aspect. We make it very explicit right now. What we are querying here is not about the individual data products, but it's about metadata. Through that same interface, we can query data, which is a bit slow. I can also query metadata. We see, this is like the Claude way of showing us it's using the discovery system data product here. Here I first ask, what tools do we have available? I think most important, if I ask what tools do you have available, I see specific tools being made available to the data product. Ideally, I can also see this one. This is the filter tool. I can interact with the system through a particular filter. Here we ask the same question, and we get all the data products being available here. We can also ask what domains are being available. Here, this is a question I was just typing in the other chat. It's like, what is the metadata? I can actually base a lot of decisions also based on metadata. Then, last but not least, I can also query actual data. Here we see it is choosing the relevant tools being offered by a particular data product through the router. Then this is being made available. This is then just the LLM behind it. It generates a nice output for it. I think this is one of the key aspects. By configuring our mesh as a single endpoint, I have progressive access to all the different data products and tools being made available there. I as an end user, I can chat with all data which is accessible to me. Of course, I only get those data products and tools I have access to, based on that. Key Insight If you want to take something away, it doesn't need to be that particular implementation, but keeping in mind domain-centric GenAI and domain-centric use cases, not exposing too much information, it actually really helps you to build a scalable and safe data architecture. See more presentations with transcripts