Transcript #
Fabiane Nardon: My name is Fabiane. I work for TOTVS. TOTVS is probably the biggest tech company you never heard of. This company has been building enterprise systems for the last 40 years in Brazil: SaaS, on-premise, everything in between. About a quarter of Brazilian GDP runs through our systems. Brazil is not a small market, it's the 10th largest economy in the world. When we started building AI agents, our problem was not lack of data, it was the opposite. We had too much data and none of it was prepared to be accessed by agents. We evolved a lot in agent building in the last few years. We have now several agent frameworks, agent orchestrations, or evals. Is our data prepared to be accessed by a token-hungry, latency-sensitive reasoning loop that can fire hundreds of unpredictable queries in a few minutes? The data we have in transactional systems was optimized to be accessed by applications.
The data we have in data lake was optimized to be accessed by data analysts or dashboards. None of this data was optimized to be accessed by AI agents. Then we have another problem, because when we talk about enterprise-grade agents, we are talking about transactional systems that you usually expect to have 99.99% precision, and we are trying to do the same with agents using probabilistic reasoning. We're never going to have the same level of precision we have in transactional systems, and we have to deal with that. Probably even the way users interact with the systems are going to change when we put AI enterprise systems. In this talk, I'm not going to talk about coding agents or personal agents that you can create using OpenClaw, for example. I'm going to talk about enterprise-grade agents, and how you prepare your data, and how you provide data for these kinds of applications.
Deterministic and Non-Deterministic Computational Models #
The kind of software we're doing now, combining AI with traditional software, we are combining deterministic software. The software we did in the last 40 years was just using deterministic computational model, but now we have a non-deterministic computational model coming together. Combining these two, of course, we can have much more powerful applications. It's not a question of using one computational model or the other. It's a question of deciding which part of the software you send to the deterministic computational model, and what part you let the AI take over to build the application. The success of your application depends on knowing where to draw the line between the two computational models, what you should send to the deterministic part of the software, and what you should send to GenAI or LLM models. This decision is based on this formula with three variables, precision, security, and cost. The three of them depend pretty much on how you provide data to these systems.
Precision #
Let's start talking about precision, or how you make your agents give better answers. The first question, especially for us, remember that most of our systems are on-premise, and they are running transactional systems. You have to decide where to get the data from. Your primary source of data, of course, is the transactional systems, your ERP, CRM. The system you have can be SaaS or on-premise, but you have a primary source of data here. Just connecting AI to your transactional system can bring several other problems, because usually the systems are not prepared to receive lots of unpredictable queries, or they don't have enough power to run historical processing that you need to run to prepare your data. Or they may have bad data with noise and you need to clean the data before sending to the agent. Or they may need to execute a semantic search, and these old databases, legacy databases, maybe they don't support semantic search.
You need a vector database or something like that. There are many reasons why people decide to copy data to the data platform, and then you can do everything you need to do in the data platform, prepare your data, and then your agent can access the data from the data platform using MCP or any other data access protocol that you may use. The only catch here is that the data in the data platform is always a little delayed compared to the transactional system, so depending on the workflow or what you have to do, you have to access the data directly from the transactional system. It's not again a question of using just one or the other, it's a question of combining these two sources of data depending on the workflow you're trying to solve. This is our recommendation in the company, where you get the data from the transactional system or when you use the data platform.
You should reach the transactional system when you need to write data to the transactional system. You're not going to write data in the data platform and then send to the transactional system. If you need write operations, you go to the transactional system. When you need fresh data, if you can't live with the delayed data you have in the data platform, then you have to reach the transactional system. Or when you need to fire some business rules that are coded in the transactional system. On the other side, you should go to the data platform when you can use stale data, when you need to process historical data, when you need semantic search that you can provide in the data platform and you may not have in the transactional system. When you need to enrich your data, it's very common to combine external sources of data to make your data better for precision purpose. This is how you decide where to get the data from.
Let's talk about how we can prepare your data platform for the agentic era. In the last few years, many companies decided to invest in a data mesh architecture. Data mesh is a very interesting architecture for especially when you have lots of data in several departments in the company. Because in data mesh, you divide your data, you distribute among domains. Each domain is specialized in a particular business domain, particular kind of data. Then you have a self-serve data platform that they all use, so the data analysts in each domain can worry about the business and prepare the data to serve the business, and they don't need to worry about the data platform itself. This kind of architecture became very popular and solved lots of governance, especially governance problems for companies trying to organize their data. Data mesh, you have a very important concept that is crucial for your data architecture.
That's the concept of a data product. A data product is a piece of data that's prepared to be accessed by others. It needs to have a data owner, that's someone that's responsible for the quality of data for maintaining that data. It needs to have a stable interface contract because you're not going to change the schema of data you are sharing with the others. It needs to have good documentation, discoverability, and quality SLA. A data product is like a microservice for data. It's a very good way of organizing the data in your company and sharing data with the others.
When we started designing our MCP tools, we decided to design our MCP tools to retrieve data with the data product. Each tool is part of a data product. This means that every tool also needs to have an owner, a stable interface contract, documentation, discoverability, and quality SLAs. These tools combine very well with the data product. If you do this, you get all the governance you get from data mesh just by doing that. Because now you have all your tools organized in your data mesh by domain. You know who is the owner of that tool. The data platform that supports that can give you statistics. For example, if this tool is still used, how it's used, how it performs, and so on. Because when we have too many tools in a company, this can become a nightmare. Who is the owner of these tools? Who is responsible for it?
Since we have all this in data mesh and in our data mesh architecture that's supported by our data platform, if you connect the MCP tools to the data product, you have this governance automatically just by using the same architecture. Now instead of doing generic tools like get_schema and generate_query, we have tools that know the data and know how to retrieve the data in a very precise and more business-oriented way. Instead of having generic tools, we have specific tools that can retrieve the data. Since the tool owner knows the business data and all the rules, they can create the best tools for each data product. Even if you do that, there's another problem that's common in every company, and it's a semantic problem. If you go to any company and ask, what is an active customer, for example? Marketing is going to say something. Finance is going to say something else.
It's very hard to get to reach a consensus. I worked for a company once that they stayed in meetings for a week just trying to define a single concept of churn that everybody could agree on, and they never reached an agreement. The problem is not having multiple definitions, it's pretending there's only one. This is not going to happen. Since you have these multiple definitions, this can work well with humans. If you are talking to someone from marketing and you're talking about an active customer, you can understand being a human and working for that company for quite some time what an active customer means for that person. When you're doing this with LLMs and the AI agents, they don't have this sensibility. You need to give more information to the agent so they can understand better the concept you're talking about.
Fortunately, there is a very old technology that can help us with that, and it's called Semantic Web. In 2001, Tim Berners-Lee, the father of the web, proposed a set of standards to give meaning to the information that you could find on the web. While I was preparing this talk, I was rereading the article that Tim Berners-Lee published in the Scientific American magazine in 2001, where he explained the concept of Semantic Web and how this works. If you read the article, he described a world where agents can talk to each other and understand each other, and they could organize and solve people's problems. In this case, the agent was trying to schedule a medical treatment. When I read this article again 25 years after it was published, I was very surprised that it took us 25 years to get to a point where you can actually use this technology to solve the semantic problem.
In the article, he says that the agent will know all this without needing artificial intelligence in the scale of Star Wars's C-3PO. He was a little too optimistic, because it took 25 years for us to get here. Now we can use this technology in our favor. The good news is that since it took 25 years to get here, the current LLM models were trained in 25 years of examples of these technologies, and they can understand the technologies proposed for the Semantic Web very well, which helps us a lot. What is the Semantic Web? Tim Berners-Lee proposed a standard called RDF. The idea of RDF is that you give a unique identifier for each concept you have. In my example of active customer, you would have one unique identifier for the active customer in marketing, and another identifier for the active customer for finance. You can have multiple concepts with the same name, but they would have a different identifier, and this way you remove ambiguity from the concepts.
Then you have standards for representing ontologies. The ontology can represent the relationship between the different concepts. Using ontology and RDF, you can have the semantic of the concepts. If you give this to LLM, they can create a context graph for AI agents, and then you have the semantics of your data. I did my PhD in Semantic Web Technologies for improving inference in intelligent systems, and this was 20 years ago. When I was doing my PhD, the biggest problem was to create the ontology. It was a lot of work to create an ontology for a domain. Now, if you have documentation about your data, and you're using LLM, it's very easy to create an ontology. That's one of the reasons why it took us so long to actually use these technologies.
Out of science fiction, let's see how this works in real life. This is an example I did with plain old ChatGPT, no special training or anything like that. I gave to ChatGPT three pieces of data. I said that Acme bought 10 servers, Beta bought 50 licenses, and Gamma bought 20 keyboards. Then I asked which customers bought hardware. The LLM with the knowledge it has said that Acme and Gamma bought hardware, because they can infer that server is probably hardware. If I give ontology to ChatGPT, and I just say — OWL is one of the standards in the Semantic Web to represent ontologies — consider this OWL ontology when answering the following questions. In the ontology, you can see that I mapped the relationship between the concepts, and I said that server is a subclass of a cloud service. I'm saying to the LLM that this is not a hardware, it's actually a software, it's a service.
Then when I gave the facts again to ChatGPT, instead of just saying it bought servers, I used a unique identifier of the concept. I'm saying that Acme bought 10 servers, but this server I'm talking about is the server with this RDF identifier here. Then I asked the same question again, which customers bought hardware, and the answer ChatGPT gave me is this one. You can see ChatGPT could understand very well the ontology and give me the right answer. In the last few years, there were several studies comparing using this kind of technology with LLM, and how this could improve the precision. For example, this study here, this is from 2024, and they proved that adding an ontology-based semantic layer improved LLM response precision by 40%. There are more recent studies that could prove even higher correctness, but it depends on how you format your data or your domain.
It's a very good rate when you compare to just using plain LLM with your data. Now you have your data products with semantics connect to the MCP tools, and doing this, it's interesting because I can return the semantics of the data based on my data product that I know very well. If I connect the MCP tool to the data product, I can return just the part of the ontology that is related to that data product, because ontologies in a big company can become really large, and you don't want to send a whole ontology to the context, so you filter the part you need.
You probably noticed in the last few years that the data platforms became much more a toolbox instead of one single tool. A few years ago, if you had a data platform, probably your data platform would run only Apache Spark, or they would be based on Athena or Hadoop, if you are old enough. Now you can see that several data platforms are offering different types of tools, and this makes sense when you're talking about AI agents. For example, our data platform has three layers. A high latency layer based on Apache Spark for batch processing of Parquet files. This layer can handle large data volumes with less cost. Then we needed also a medium latency layer that's based, in our case, in Google BigQuery, that can also handle large volumes of data, but it has more cost, because BigQuery, if you have an infinite credit card, which is not in our case, it can be very fast, but it can be very expensive.
For agents, what we needed to do was to provide a low latency layer in the data platform, in our case based on Postgres and DuckDB, that can handle smaller data volumes with less cost, but everything is orchestrated by a unified processing interface in our data platform. The reason for that is that you need to have something unified, so you can create your data pipelines in Spark the same way you create your data pipelines in BigQuery or in Postgres. We have a unified interface for that. You probably noticed that some data platform vendors announced recently, I think Databricks announced that they are going to support or are supporting Postgres as well. We've been doing this for two years, more or less, just because we needed to serve agents, and we need to have a low latency option. Why do you need low latency when talking about AI agents? This is interesting because before, when you're talking about data platforms, usually you had your data in the data platform to do historical processing, and your data analysts, data scientists can create dashboards so they could use this data instead of going to the transactional systems. Now that you are providing data for agents using your data platform, you need something that is different. You still have to do your historical processing in the data platform to prepare your data, but you need to have a different latency when serving this data. When we talk about low latency in the data platform, you need two kinds of low latency. The first one is low latency in processing. You need to make sure that the moment you receive the data in the data platform, to the moment the data is ready to be served, this time has to be very fast.
You need to get the data, clean the data, reach the data, and make the data ready in the shortest time you can because you want to increase the freshness of the data. Of course, you're not going to be able to do that with all data processing you have. This makes sense for data processing that you can do really fast. For the data, you have to do a huge processing, a very complex processing. You may do this processing using Apache Spark, for example, but then you need to make the data available to be retrieved by agents in a low latency level. There are two kinds of low latency. Low latency in processing and low latency in data retrieval. When we were designing our low latency layer, we wanted something that was vendor-neutral, so we could use in multiple clouds. We wanted something that was low cost, that was fast, reliable, and that would support semantic search.
We needed to also make sure that when you get the data in the first step here, when you get the data, this would fire the transformation pipeline immediately. You'd have this pipeline running really fast. We wanted to make sure that this pipeline was atomic, meaning that the data would be consistent. I want to make sure that as soon as I get this data here, this would be fired, this would be fired, and I would have the data ready. We are talking about old technologies. There is a very old technology that can solve this problem, and it's called a transactional database. We used Postgres to solve the problem. When I get the data in the first layer here, in the raw data, this fires a trigger that executes a series of stored procedure. I know it's not fancy, but it works very well. I make sure that the transaction is going to end, and my data is going to be right in the end of the pipeline.
This pipeline can run in milliseconds or seconds, depending on what I have to do. Using Postgres and combining with the other technologies in the data platform, I can do very interesting stuff. For example, I can have BigQuery data on Parquet files, combine this data using Apache Spark, save the data in a Postgres table, and everything is transparent to our users because I have a unified processing interface. If you go to our data platform, you can create as many data pipelines as you want, and you just say which engine you want to use. Then you can combine the data to make it available to be used by the agents in the end. Using that can combine different processing methods. The low latency layer is prepared to answer the agent questions and support semantic search through vector database in the Postgres database. Now I have the data product with semantics, a low latency layer, and the MCP tools over it. Tools are actually just predefined Postgres queries and the data platform provides also tools to create tools in the data platform. All the tools are not programmed by a separated tool or system. They are created inside the data platform using our Postgres database to provide access in a low latency form.
Security #
Let's talk about security. There's a very popular approach when retrieving data from databases, which is just create a tool, a generic tool that can read the schema and let the LLM write the queries. This, of course, is very flexible, but also vulnerable to prompt injection. There's a more secure approach, that's the approach we are using, that is to create tools with parameters to retrieve data with the security embedded in the tool code and not letting the LLM to generate the code. Of course, this is less flexible because the only data the agent can access, is the data that has a tool that's able to retrieve it. On the other side, it's less vulnerable to prompt injection. Even if I do that, for several tools, I need to know who is calling the agent. I can retrieve only the data that that particular person can see.
What we do is something like this. The AI agent goes to the company identity provider and logs in. Probably there's a human in the loop in this case. The company identity provider returns the logged user. The AI agents, when they call the MCP server, the AI agents are going to authenticate using OAuth. In the OAuth token, you have the logged user. You can propagate the logged user to your tools, and then using this information to filter the data you need. Here I'm going to retrieve only the employees that this logged user is the boss. This way, using identity propagation, I can send this information to our tools. This is done also automatically by our data platform. When you invoke our MCP server, you're always going to get the logged user that we got from our company identity provider.
Cost #
Let's talk about the last variable, which is cost. There's a famous saying in Brazil by an Austrian writer that say that Brazil is the country of the future. We Brazilians like to add with a little bit of humor, and it will always be. There's one aspect where Brazil is living in the future, and that's token economics. You see, when you say that Claude Code costs only $100 bucks, for us it's $500 bucks. This is a third of the Brazilian minimum wage. The cost of tokens hits us way before it hits you. I see discussions about cost of tokens emerging in the last few months. We are living this for years. Which is good, because scarcity makes you more creative. We had to deal with token costs also. Data has a huge impact in token costs. If you use more precise tools, return less useless data, and especially push logic into the tool and not to the prompt, you can save lots of tokens.
What we decide to do is to create multiple MCP servers for the agents. Then we had a problem that you may have one particular MCP server that's called once, one particular MCP server that's called many times, and several MCP servers that are never called. There's no way to know this up front because you're creating an agent and you can't predict how people are going to use this agent, what calls are going to be made. What we did was to implement a pattern that we created called MCP Fabric. The idea is that instead of having deployed multiple MCP servers, you have just one service that can serve several virtual MCP servers. In our case, we did this using Spring AI. Each MCP server is just a URL that is available in a single service, so you can create hundreds of MCP servers without having the cost of deploying a service for each of them.
With this, this is the final architecture of our agentic data platform. We have different latency layers, the high latency, medium latency, low latency layer. A unified processing interface that organizes everything so you don't have to worry what is running behind it. You have, in the data platform, the concepts of data products, semantics, a tool creator, and an MCP creator. Everything is created inside the data platform with MCP Fabric to support many MCP servers, OAuth, identity propagation to solve security. It's easy, fast, and very cheap to create new tools and MCP servers. You can make tools return what the LLM needs and reduce token consumption. Then we can have a huge problem that is having too many tools in the context. Of course, this is going to cost more tokens. This is how we solve it. We have a layered token economy. First, we decide to create one MCP server for each agent.
Since it's very cheap, very easy, and very fast to create an MCP server, I can create one for each agent. This MCP server is going to have only the tools that particular agent needs. Then we have dynamic tool search. Let's see how this works. For each agent, I'm going to create an MCP server. If you used MCP gateway, this is pretty much the same. When you have an MCP gateway, what you actually do is to select the tools that are going to be available in this virtual MCP server. You're going to give to a particular agent an MCP server that has only a few tools you need. We did the same, but we don't use the MCP gateway. The data platform is the MCP gateway. We can create an MCP server for each agent. We have a subset of the available tools for that agent.
Even in this way, I can have a problem because I may have one particular agent, one particular MCP server that has many tools, and this can fill my context very fast. To solve that, we have also the dynamic tool search. The idea is that the agent is going to call the MCP server, ask for a list of tools. The MCP server is going to send all the tools available with the descriptions and everything. Instead of putting in the context, the agent is going to send these tools to the search tool service. The search tool service, they are going to have these descriptions. When the agent needs something to solve a problem, it's going to call search_tool in the search tool service. The search tool service is going to return the tool's candidates to solve that particular problem. It can do this using semantic search, or Lucene, or regular expressions, depending on the problem you have.
Then you have the five tools in the context. Then the agent can call the tool. We did a benchmark using dynamic tool search. You can see that this is the number of tokens you used with 10 tools, 25 tools, 50 tools, and 100 tools. The orange bar is using dynamic tool search, and the blue bar is just putting all the tools in the context. You can see, you can save lots of tokens using an approach like that. Another thing you can do to save some tokens is formatting your tool response to use less tokens. Usually, JSON is the best especially for precision. It's very easy to use. It's very easy to export. It's very good to represent nested structures. It's also very verbose, and you use lots of tokens. If you have only flat data, and you return in CSV, you can save up to 50% tokens in your tool response.
There's another standard we are testing, which is pretty new, it's called TOON. That looks like this. TOON can represent also mixed tabular data and complex data, and can save 30% to 60% tokens in the result. The problem of TOON is that since it's very new, sometimes it doesn't give you the same precision because the LLMs are not trained in years of TOONs, they are trained in years of CSVs and JSON files. There are ways you can use it to improve precision, and that's something we are testing now.
Summary #
Here's a summary of the techniques. Some can be used to improve precision, some are going to solve security issues, and some are going to be used for improving the cost of token. All of them can help you provide better data to AI agents. If you push your code to the deterministic part of your software, it's going to be more precise, more secure, and more cost efficient. On the other side, the magic happens in the non-deterministic part of the software. It's where the agent can understand natural language, reason over incomplete information, navigate ambiguity. The success of your software will depend on knowing where to draw the line between these two computational models, and provide data for the agent. It's very important for making the magic happen.
Questions and Answers #
Participant 1: Is there a possibility that we could use more tried-and-true serialization mechanisms? Like, for example, as long as you have the schema, for the semantic schema, couldn't we use a Protocol Buffer type of protocol to compact the data and still retain all the goodness that we get with JSON?
Fabiane Nardon: Yes, it depends. I never tested Protobuf or something like that. It depends on how the LLM is going to be able to interpret and give you the right answer. For example, comparing TOON to JSON. The LLM usually can understand better JSON with less ambiguity than TOON. Depending on the other protocol you're using, how good the LLM is going to be in understanding that standard and give you the right precision. That's something we are experimenting right now and trying to find a middle turn between having precision and having cost economy.
Participant 2: You talked about RDF and uniquely identifying entities in your ontology. I'm curious, have you dealt with the problem of deduplication in representing your semantic ontology and how have you tackled that?
Fabiane Nardon: Deduplication of data in the ontology, is that it?
Participant 2: In your graphs as well.
Fabiane Nardon: In the graph? Of course, it's not easy to represent the RDF and the ontology. This is an art as well. Depending on how you do it, you can have the same concept with different identifiers as well. Since we are organizing our data in domains and in data products, this can be not eliminated, but we can reduce it a lot, because you have each responsible for a domain creating their part of the ontology that are going to be connected to the bigger ontology. Of course, this is not perfect, but we can solve the problem a little bit by organizing in domains and data products.
Participant 3: I saw one of your slides where you specifically talked about your implementation of the MCP searching functionality, and I found it very fascinating that on your main orchestrator chat that you were representing, that the context was not polluted at all when you initially searched the first 100 tools. Then there was a secondary call that refined that down, I think, to like five tools or something, and that was the thing that got passed over to your main agent. Is it possible for you to talk a little bit about how you're doing that initial 100 search without polluting your context window at all?
Fabiane Nardon: Maybe I oversimplified because, of course, there is one tool in the context that is the search tool? What we did is we implemented using Spring AI as well. You actually give to the agent just one tool, that's the search tool. Using Spring AI, you can inject two references in the context. When you call and you get the five tools back, you just inject those five tools in the context.
Participant 3: What was the 100 tools to the 5 tools?
Fabiane Nardon: The 100 tools is when you have your MCP server and when you call tools list, asking for the list of tools that are available, the MCP server is going to return a list of tools. It's a representation of all the tools and the descriptions of the tools. It's usually what the MCP client is going to get when they communicate with the MCP server for the first time.
Participant 3: I think I misunderstood. You're essentially saying that search tool tool is its own agent and that's staying in its context, and then the five tools returned is what's going.
Fabiane Nardon: Yes. You have two services, in the search tool, you have all the tools, and in the context of the agent, you have only the tools that you got from the dynamic tool search.
Participant 4: I love how you made it super easy to create tools on top of your data platform. Who usually creates the tools? Is it the data product teams or the agent teams who will be using it?
Fabiane Nardon: It's a mix. Usually, the data team knows the data better and they create the tools. Usually, the teams that are doing the agent, they also are able to use the data because the agent, you need to know the data you have, especially in our business. It's different because in our business, we're creating enterprise agents for our systems. If you are creating, for example, tools for GitHub and you're going to let anyone access your tools, it's different. In our case, the people that are creating the agents, usually they have access to the data. Because sometimes the one group, one domain created tools to retrieve data and the other team is just creating an agent and getting a piece of data from another domain so they can reuse the tools. In this case, you had one team doing the tool and the other team doing the agent. It's a mix.
Participant 4: Both, like either data product teams, they can create tools, but agent teams, they have also this ability and instrumentation?
Fabiane Nardon: Yes, the agent team can create tools as well if they have access to the data. Sometimes it's the same team, sometimes it's not. There are also tools that are ready from other domains. Either if you need that tool from that other domain, you can just access the other tool or you can ask the data team from the other domain to create a specific tool for you. You create agents for your product and you can combine tools from several different domains. It depends. The data team is usually the one that creates the data tools because they know the business and the data schema very well. Usually, this is the way it happens.
See more presentations with transcripts