# What is a semantic knowledge graph, and does your AI need one?

> Source: <https://www.cio.com/article/4249101/what-is-a-semantic-knowledge-graph-and-does-your-ai-need-one.html>
> Published: 2026-11-18 10:00:00+00:00

**Recommendation:** Give agents a semantic knowledge graph of business relationships and meaning, not just raw tables, so they can trace a delayed order across 15 departments and name the holdup.
More in the playbook "Designing agent systems": <https://wpnews.pro/playbooks/agent-architecture>

A traditional database organizes information into tables. One column is the name, another is the company they work for. A graph database stores the same information not in tables but in nodes and edges. A person could be one node, the company they work for is another, and the edge is the connection between the two.

Any node can be connected to any other one, making graph databases much more flexible. And finding related entities is as simple as following the edges from one node to another. They’re useful for navigating supply chain dependencies, detecting fraud, or mapping user preferences to make better recommendations.

All of that is already potentially useful to AI agents, but that’s not all. A knowledge graph sits on top of the graph database and is a map of the relationships. The node with the person’s name, for example, could be connected to the node with the company information because the person is an employee of that company. The knowledge graph helps explain what the data is.

A semantic graph, or a semantic knowledge graph, takes this one step further and adds a deeper level of meaning. For example, the list of companies could actually be a list of clients or potential clients. The list of people could also be potential contacts for sales calls, and their job titles could indicate their relative importance.

The semantic layer is all about what the data means for the enterprise and how the data is actually used in practice. When AI agents are assigned actual work tasks, they need to have this information.

“Agentic AI is AI that can do things, not just tell you things,” says Rafael Domene, global CIO at manufacturing company Cosentino. “But you have to provide what AI needs to do the job in the right way. Not just data, but information. It’s about business knowledge and understanding what’s going on.”

Cosentino had already invested a lot of money into a massive data lake and data lakehouse, as well as ERP and CRM systems. “You can collect all the data, but that’s not enough to explain what’s going on,” he says. “If a KPI goes red, you need to dive deep into the organization to understand what happened.” Traditionally, that means human experts with a deep understanding of how the company works. So how can this process be automated with AI? The semantic layer gives AI the business context behind the data.

Historically, building this semantic layer has been a tough job. It requires subject matter experts, and it’s not a one-and-done project since the way a company does things changes constantly. It usually wasn’t worth the effort, especially if the analysts looking at the data already had the context to understand what it meant due to their knowledge of the company. They didn’t need a semantic layer because they had their own human understanding.

AI agents, however? Not so much. Not only do they have a bad understanding of how companies work in general, but they also don’t know much about how any one company works in particular.

Cosentino began building its semantic layer three years ago and aims to be finished by the end of the year. “The main processes and flows are already there, and the remaining work is mostly on the financial side of the business, including logistics, manufacturing, operations, customer service and sales, procurement, and HR.

The effort is already showing results, he says. Take for example a delayed order. “Service levels for our customers are really important,” says Domene. “But there are 15 departments involved in the process and all of them had their own dashboards. We were having a lot of issues finding out last minute that a specific order wasn’t going to be served.”

Now an AI agent can look at any order, for any customer, and predict whether it’ll arrive on time or be delayed. And if there is a delay, where the holdup is.

“If the raw materials don’t arrive on time, or the factory has an issue, the process isn’t going to reach the customer on time,” he says. But if the salesperson knows about the problem as soon as it happens, they can go back to the customer, explain what happened, and offer potential solutions.

“It’s not just about providing information,” says Domene. “Without context, I can’t link things. What happens with this process, why is this order different from the previous one, why is this happening, what can I do, and what’s the best course of action so we can make the right decision for this customer or product?”

Or, say a customer exceeded a credit limit and an order was blocked. In the past, that would result in an average delay of 22 days. Today, the maximum length of delay is one day.

“This is a service and performance improvement,” he says. “There was a team of people in the finance department doing those things. And now they’re free from it all, doing things that are more vital for our customers.”

According to Domene, the company saw up to $10 million in savings in the first year of the project. “The return was really fast,” he says, “and the benefits are way beyond more savings. It also impacts our revenues.”

By the end of the year, he says, the company expects to get to $100 million in savings. And there are other benefits. For example, sales employees used to spend more than a third of their time on administrative work, and they’ll soon have AI agents assisting them in those tasks. Cosentino began building those agents last year, testing them this year, and expects to have them fully operational soon.

“Not all of that saved time is going to be turned into revenue,” Domene says. “We’ll see how much we get out of that, but we see massive potential.”

When the knowledge graph and semantic layer process began, it was largely a manual process, managed by the company’s own expert employees, and combined with a group of engineers from the vendor the company chose for the project, process mining company Celonis.

“Systems like SAP have fixed flows and context so it was relatively easy,” Domene says. “But it still required effort by humans.” Then, over time, AI was added to augment the humans.

Still, the process of building the semantic layer isn’t yet ready to be fully automated.

“Humans don’t always create documents in ways the AI can understand semantic relationships,” says David Hoelzer, a fellow at the SANS Institute, and chief of operations at Enclave Forensics.

AI can help accelerate the process of building the knowledge graph, he says, and it can help keep it up to date. But it’d be a mistake to let the AI try to build completely on its own. “It may end up with relationships that don’t make sense,” he says. “You still need people and they need a very particular set of skills.”

About a year ago, the standard way to feed new information into an LLM was via RAG in combination with vector databases. A company would take the information that could be useful to an AI, and convert it into a vector database that takes unstructured information and organizes it so similar topics are grouped together. That way, when a prompt is sent to the LLM, it can include all the relevant context.

It’s still a good approach for unstructured data for old-fashioned RAG pipelines, but it tends to result in a lot of material, some of which can confuse the LLM or be dated, and excess content means more tokens, which means higher costs. AI agents tend to make a lot of calls to LLMs, so it’s a bad idea to be too wordy. Plus, knowledge graphs allow agents to find the exact information they need, while the semantic layer gives them the knowledge about what to do with that information.

New York Life Group Benefits Solutions uses both vector databases and knowledge graphs. The knowledge graph it uses is built on top of the LangGraph AI agent framework from LangChain.

“The knowledge graphs work for multi-agent orchestration and the vector database for gen AI RAG applications,” says Matt Marze, the company’s CIO.

The knowledge graphs begin with the basic business information, data, signals, and external data sources, such as information about the broker community. Then the semantic layer includes business understanding, relationships between the data, and common definitions around products and plans, in addition to, he says, the business context of the current situation, history, workflow, and business priorities. “All that drives into insights, predictions, and recommendations, and allows us to coordinate the abilities of the AI agents,” he says.

As agentic AI projects proliferate, the interest in knowledge graphs continues to increase.

“A year and half ago, I was talking about it and people looked at me like I had three heads,” says James Kaplan, distinguished partner at McKinsey & Company. “But in the past year, we’ve seen real uptick for knowledge graphs. Many institutions I know of realize they can use knowledge graphs to support more complicated decision making, and in conjunction with agents, enable more capable agents.”

Accuracy and hallucinations are a big problem for AI models in general, but in agentic systems with multiple calls to an LLM, inaccuracies can quickly compound.

“All the models are probabilistic ones,” says Zakir Hussain, partner and EY Americas data and AI leader for industrials and energy. “As soon as you try to use any of those models to answer enterprise-level questions, or questions you want to base your decisions on, they start hallucinating because in many cases they don’t understand or misinterpret the context.”

For business applications in particular, the inaccuracy issue can be a big one, since AI models are trained on generic information. “As soon as it becomes specific to a company, that’s where things start going sideways,” he adds.

According to a [paper](https://arxiv.org/html/2502.13247v3) released at the end of 2025 by researchers at UC Santa Barbara and JP Morgan AI research, knowledge graphs improve accuracy by 26% or more over standard chain-of-thought baselines.

“This whole conversation around ontologies, semantic layers, and knowledge graphs is pretty important right now,” says Hussain.

Every organization is trying to figure out how to leverage AI for growth, efficiency, and productivity, he says. “To be able to unlock these three layers — growth, efficiency, and productivity — you have to codify the context of the company.”
