Chief Scientist, Neo4j
4 min read
Agentic systems mark a new phase in the evolution of artificial intelligence. AI agents now navigate and act in the world. Their potential is breathtaking, but so are the risks they pose. A hallucinating agent doesn’t simply waste someone’s time. It files the wrong ticket, books the wrong meeting, or pays the wrong invoice. Every AI strategy needs to address this reliability issue—it’s now the greatest hurdle facing enterprise AI.
The UK’s National Innovation Centre for Data (NICD) recently compared the two leading techniques for improving agent reliability and found that vector + graph RAG, or GraphRAG, dramatically outperforms vector-only RAG. Agents using GraphRAG were 80% more “truthful” and answered over twice as many questions, while using tokens more efficiently.
The research was sponsored by Neo4j but conducted independently by NICD.
GraphRAG generates accurate, highly relevant answers without wasting tokens #
NICD found that GraphRAG improved both precision and recall in factual accuracy while reducing hallucinations and improving truthfulness scores. And it did so while using tokens more efficiently than vector-only RAG. Here’s a look at the key findings.
Fewer hallucinations
The NICD team broadly defined “truthfulness” as the difference between a system’s accuracy and hallucination rates. They evaluated the RAG approaches using a fine-grained truthfulness score, which penalized hallucinations while accommodating partially correct answers. GraphRAG achieved a score of 63, significantly outperforming vector-only RAG, which scored 35.
More precision and better recall
GraphRAG significantly increased both the precision and recall of the AI system’s factual correctness. The GraphRAG configuration achieved more than twice the precision and recall of the system that relied solely on vector-only RAG. (GraphRAG scored .38 vs. .18 for precision and .35 vs. .15 for recall.) This mitigated the common issue of mixed false positives and false negatives.
Agents using GraphRAG vs. vector-only RAG
• 80% more truthful
• 2x more questions answered
• Fewer hallucinations
• More efficient token use
Greater willingness to answer
Vector-only RAG implementations tend to be conservative, often issuing a “safe refusal”—saying the answer is unknown—when they lack the necessary context. Vector-only RAG attempted to answer only 28.9% of complex questions, while GraphRAG successfully answered over 65.3% of the questions, more than halving the refusal rate.
Improved cost-efficiency
The GraphRAG system included specialized graph tools that could target specific section headings and infoboxes, so the agent could avoid reading entire articles unnecessarily. This significantly reduced the number of tokens needed to generate a correct result, increasing relative value per token. The GraphRAG system only consumed tokens that directly contributed to highly accurate, factual responses.
It’s worth reading the complete NICD report, which offers more detail on the benchmark dataset, retrieval tools, answer quality metrics, and more.
AI you can depend on: New independent research #
NICD tested three AI configurations on 510 complex questions.
Improve agent reliability without a major data project investment #
The NICD researchers built an agentic system that included vector-search and graph-query tools operating over a Neo4j knowledge graph. Importantly, they didn’t spend weeks or months building an elaborate hand-engineered ontology. They built a lightweight graph structure from the kinds of documents most businesses already have.
In this case, they used the titles and sections of a “significant proportion” of Wikipedia articles, as well as the links between them, proving that organizations can improve agent performance without taking on expensive, time-consuming data projects.
NICD tested the system across 510 complex questions, some of which required dozens or hundreds of steps to solve, and used a demanding QA benchmark (MoNaCo, from “more natural and complex” questions) to evaluate system performance.
The researchers measured the system’s accuracy and token usage when it had access to a vector database, a vector and graph database, or neither. “The objective of a strong system,” they note, “is to maximise answer accuracy while minimising token usage.”
Learn more about GraphRAG and agent reliability #
GraphRAG has become a critical tool for ensuring agent reliability and helping organizations see a return on their AI investments. By combining knowledge graphs and vector search, GraphRAG provides agents with deep context and multi-hop reasoning for more accurate, relevant, and explainable results.
To learn why vector databases alone can’t ensure enterprise-grade reliability for AI agents, and how GraphRAG delivers the accuracy they need, take a look at this explainer video.
For a more detailed look at the components of GraphRAG, read this post. And if you’re interested in how GraphRAG solves real-world data challenges, check out these use case videos: