Graph database supplier Neo4j says graph tech can stop AI models and agents dreaming, hallucinating, and fabricating false answers to questions.
It pointed us towards a recent academic paper; ”Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation" published on arXiv in June, that it says proves its point.
The paper’s abstract states it explores “the idea of using a lightweight graph structure with a relatively simple graph schema, to support the RAG (Retrieval-Augmented Generation) subsystem via a dedicated toolset. We design an agentic system with a variety of vector search and graph query tools operating over a structured dataset based on a curated subset of English Wikipedia articles, and evaluate its performance on questions from MoNaCo, a challenging Wikipedia QA (question answering) benchmark of complex query answering tasks.”
They pose a question for an LLM to answer: “Can you name all the battles between the Dutch and English in the First, Second and Third Anglo-Dutch Wars, and list the victor of each battle?”
Answering this requires “a sophisticated retrieval and reasoning process. In fact, it requires multi-entity and multi-hop reasoning, and cross-document access, all at once.” They observe that: “These types of questions pose a significant challenge to current state-of-the-art LLM-based systems.”
They asked the question to three kinds of LLM and evaluated the results:
-
Vector+graph RAG - using a unified vector and graph database with a series of pre-defined tools to improve retrieval from external knowledge bases (KBs).
-
Simple vector RAG
-
Zero-shot LLM with no RAG
The paper says: “The results shown indicate that augmenting a basic vector RAG subsystem with a simple graph-based KB and corresponding tools can significantly reduce the amount of hallucinated content” but not completely: “(the coarse truthfulness score improved from about −127 to −49 for vector+graph RAG vs zero-shot).”
But it’s much better than vector RAG alone: ”We also show that, when partially correct answers are taken into account, vector+graph RAG achieves the highest score across all three evaluated scenarios; the fine-grained truthfulness score was 80 percent higher than for vector RAG. Additionally, the factual correctness results indicate that vector+graph RAG achieves more than twice the precision and recall of the system based solely on vector RAG.”
All-in-all: “By increasing both precision and recall while reducing hallucinations, the proposed solution is a promising direction towards increasing trust in LLM-based QA systems.”
Read the paper for a detailed look at what's involved.