Chief Product Officer, Neo4j
6 min read
A fast-food restaurant’s AI agent fails to identify an obviously erroneous order, leading to comically massive over-ordering. News of the mistake goes viral.
A coding agent deletes a software company’s entire production database, preventing multiple car rental companies from reserving and assigning vehicles for a day.
A health insurer’s agent overrides doctors’ recommendations and denies post-acute care to elderly patients.
These examples of real-world agent failures come from a new MIT Technology Review Insights report, “Connecting AI agents to enterprise knowledge.” The authors identify a “curious shortcoming” in many agents: Despite consuming immense amounts of data, they don’t understand how businesses operate.
According to the report, agent ignorance is likely to have played some role in the failures described above, and it’s a major reason so few agentic AI projects succeed. The MIT researchers surveyed 300 data executives and found that only 34% of agentic projects make it to production. AI experts quoted in the report think the figure falls below 20% when organizations need a high degree of accuracy.
Yet a select few organizations successfully deploy 61% of their agentic AI projects. These “production leaders,” just 13% of the businesses in the survey, have something critical in common: They equip their agents with knowledge, not just data.
Why AI production leaders succeed: Their agents understand the business #
The report found that “knowledge capabilities” strongly correlate with production success, and that production leaders had by far the most advanced capabilities. Knowledge capabilities supply agents with three kinds of knowledge:
- Semantic knowledge – how an organization broadly functions and what it values: key relationships, concepts, rules
- Procedural knowledge – how an organization executes important actions
- Episodic memory – the record of an agent’s own behavior: past decisions and actions, and their outcomes
When agents have all three, they grasp organizational goals and priorities and know how to do things like debug code, approve or deny a loan, and book a flight. They also learn from every action and mistake.
This knowledge significantly mitigates AI risk. Understanding the proper procedure, fast-food agents would double-check order volume, and coding agents would protect production databases. Knowing that a physician explicitly recommended a course of treatment, a healthcare agent would approve it.
Many executives, the researchers found, recognize that agents don’t just need data—they need knowledge. More than four in ten (44%) thought strengthening the “structural foundation” between organizational data and agents would improve agent performance. Of those, a majority (57%) thought superior agent knowledge would increase operational efficiency, 48% thought it would improve productivity, and 41% thought it would reduce the risk of incorrect decisions.
KNOWLEDGE FUELS AGENT SUCCESS
• 34% of agentic projects reach production
• 13% of businesses are AI “production leaders”
• 61% of production leaders’ projects succeed
• 62% of leaders provide agents strong semantic knowledge
• 21% of non-leaders provide it
How a knowledge layer brings meaning to agents #
According to the report’s AI experts, building a knowledge layer is the best way to give enterprise agents the knowledge they need. A knowledge layer allows agents to understand organizational data, access and analyze data relevant to specific tasks, and learn from new data and their own behavior.
The components of a knowledge layer map to these capabilities. A knowledge layer stores data as a graph, with an ontology that provides a live map of the enterprise by defining the meaning behind business concepts, rules, and relationships. This is the semantic and procedural knowledge MIT describes. Agents can rapidly traverse relationships between entities in the graph, pulling the precise subset of organizational knowledge—the exact context needed for their task—to execute multi-hop reasoning and make accurate decisions. Agent memory—short-term, long-term, and reasoning memory—enables agents to learn from past decisions and improve future performance. It encompasses the episodic memory in the MIT report.
Sanjeev Mohan, principal at AI advisory firm SanjMo, told the MIT researchers that a knowledge layer bridges the gap between a business request and the technical layer, ensuring business concepts are consistently defined:
Jim Jacob, Executive Director of Digitalization, Advanced Analytics, and Artificial Intelligence at Cummins, notes that a knowledge layer unifies fragmented enterprise data: “Knowledge maps create a front-end view of structured common data. The connections go into disparate datasets, but they create a common dataset view that an agent can depend on.”
Cummins and Gilead Sciences use a knowledge layer to outpace the competition #
Jacob believes a knowledge layer is the agentic foundation most enterprises lack. He compares global manufacturing firms like Cummins to “living organisms” in which sourcing, manufacturing, products, and logistics all interact. The flexible ontology of a knowledge layer captures these connections, making it essential to agentic projects. With it, he says, agents don’t just avoid mistakes—they exceed expectations.
“The agent can say, ‘You’ve asked me for a forecast for raw material X, but by the way, widget Y, which is highly dependent on raw material X’s success, is in short supply. So procure widget Y first and then buy raw material X.’”
To Jacob, that “by the way” is the most valuable kind of intelligence agents provide Cummins—critical insights the organization would have missed.
Thomas Luu, who leads the Anti-Counterfeiting Team at biopharma firm Gilead Sciences, told the MIT researchers that his company uses a knowledge layer to identify extensive fraud networks. The most sophisticated schemes look normal, but the organization’s knowledge graph—a core component of its knowledge layer—uncovers indirect connections between key entities: storage facilities, individual pharmacists, legitimate and illegitimate pharmacies, etc.
“Our agent may then determine that an entity is one or two steps, or ‘hops,’ away from the actual fraud,” Luu notes. “They might not be directly committing it, but they’re close to it.”
An established pharmacy receiving illegitimate products, for example, might be operating within normal thresholds. A knowledge layer enables anti-fraud agents to uncover its relationship with a “burner” pharmacy or illicit storage facility.
When agents have the knowledge they need, they justify your investment #
Interviews with hundreds of data executives led the MIT researchers to a clear conclusion: Agents fail when they don’t understand your business. A knowledge layer provides that understanding, and the report ends by describing its main business benefits: trust, transparency, and scale.
When agents understand how organizations work and what they value, they make far fewer mistakes, engendering trust across the business. When they’re grounded by a knowledge graph, their decisions become fully traceable. And when they share one trusted source of enterprise knowledge, they’re much easier to scale across the organization.
Agentic failure means wasting investments and ceding ground to competitors, but there’s a simple way to avoid it: enterprise knowledge, delivered as the right context for every agent.
Why AI agent projects stall before production
MIT research reveals what the top 13% of organizations do differently.