Language Model Hallucination Evaluation with GraphEval

Amazon researchers have developed GraphEval, a framework that uses knowledge graphs to detect hallucinations in large language model (LLM) outputs. The method constructs semantic triples from a model's response and evaluates each triple against a ground-truth context using a natural language inference (NLI) model, flagging any triple that cannot be entailed as a hallucination. GraphEval emphasizes explainability by identifying exactly where factual inconsistencies occur.

Language Model Hallucination Evaluation with GraphEval Turning the key principles and methodological stages of GraphEval into a simulated practical scenario to better understand its usefulness and key implications in understanding and combating LLM hallucinations. Introduction Hallucinations are one of the best-known problems that large language models LLMs may experience when generating responses. They occur when a model produces a response that is factually incorrect, nonsensical, or simply made up, typically due to the model's lack of internal knowledge on the matter. While many solutions have arisen in recent years to tackle the problem of model hallucinations, methodological evaluation frameworks for internally diagnosing them have been comparatively less studied. One recent study by Amazon researchers proposes using knowledge graphs as a means to analyze and detect hallucinations occurring in LLMs. The framework presented in the study is named GraphEval . In this article, we will take a gentle, practical approach to illustrate the conceptual building blocks of GraphEval through a simulation-based, lightweight code example that you can easily try on your machine. GraphEval in a Nutshell GraphEval leverages knowledge graphs to identify and signal hallucinations in LLM-generated outputs. Unlike classical performance metrics that provide single scores to evaluate aspects like accuracy, certainty, and so on, GraphEval applies a two-stage evaluation process that emphasizes explainability, namely, providing insights into where exactly the hallucination took place. To do this, GraphEval considers two stages: - Constructing a knowledge graph from the generated model response. The graph consists of semantic triples of the form Subject, Relationship, Object , where subjects and objects correspond to nodes, and relationships correspond to the edges connecting those nodes. - Evaluating each triple in the constructed knowledge graph against a source context a ground-truth body of knowledge through a natural language inference NLI model. Any triple that cannot be entailed by the context according to the NLI engine — because it is contradictory or neutral — is flagged as a hallucination. Illustrating GraphEval Through a Code Example Before starting the code that simulates the application of the GraphEval framework, let's make sure we have the necessary libraries installed: pip install -q transformers networkx matplotlib torch The purpose of the code example we are about to walk through is to demystify how the GraphEval methodology works, so we will replace the stages that would demand a heavy computational burden in a real-world setting with simulated, lightweight alternatives. Accordingly, we will simulate a ground-truth knowledge base context assumed to contain factual information. In a production setting, this ground-truth knowledge would stem, for instance, from retrieving relevant documents from the vector database of a retrieval-augmented generation RAG system https://machinelearningmastery.com/understanding-rag-part-vii-vector-databases-indexing-strategies/ . For simplicity, here we directly create a ground-truth context and store it in source context . The ground-truth context provided to the LLM source context = "GraphEval is a hallucination evaluation framework based on representing information " "in Knowledge Graph KG structures. It acts as a pre-processing step and utilizes " "out-of-the-box NLI models to detect factual inconsistencies." Now, let's suppose the following is the original LLM response to a user prompt like "explain succinctly what GraphEval is". To initiate the first stage of the evaluation process, we would ask an auxiliary LLM to build the knowledge graph from that response. Both the response and the follow-up prompt used to obtain the knowledge graph are shown below: The generated response we want to evaluate contains a hallucination llm output = "GraphEval is an evaluation framework that uses Knowledge Graphs. " "It requires a highly expensive, enterprise-level server farm to operate." Prompt template that would theoretically be passed to a local/free LLM e.g. Mistral-7B KG EXTRACTION PROMPT = f""" You are an expert information extractor. Extract the core information from the following text as a Knowledge Graph. Return the output strictly as a Python list of tuples in the format: Subject, Relationship, Object . Text: {llm output} """ Once again, for the sake of simplicity and to bypass the otherwise heavy computational load of running a massive LLM locally, let's suppose the following graph triples are obtained: Simulated extraction to bypass the heavy computational load of running a massive LLM locally extracted triples = "GraphEval", "is", "evaluation framework" , "GraphEval", "uses", "Knowledge Graphs" , "GraphEval", "requires", "expensive enterprise server farm" print "Extracted Triples:" for t in extracted triples: print t Output: Extracted Triples: 'GraphEval', 'is', 'evaluation framework' 'GraphEval', 'uses', 'Knowledge Graphs' 'GraphEval', 'requires', 'expensive enterprise server farm' We deliberately added a triple that is fundamentally a hallucination no enterprise server farm needed whatsoever , so we can demonstrate how the subsequent NLI process applied to the knowledge graph reveals it. Enough simulated steps for today. Let's get into the real action for the next stage: the NLI process. The next piece of code is fundamental to leveraging the ideas behind GraphEval. It uses a pre-trained NLI model from Hugging Face — the model is publicly available, so no access token is needed to download it — to compare each triple against the ground-truth context. If no entailment is "predicted" by the NLI model for a given triple, it is labeled as a hallucination. python from transformers import pipeline Loading the open-source NLI model print "Loading DeBERTa NLI model..." nli evaluator = pipeline "text-classification", model="cross-encoder/nli-deberta-v3-small" def evaluate triple context, triple : subject, relation, obj = triple hypothesis = f"{subject} {relation} {obj}" Checking if the context entails the hypothesis result = nli evaluator {"text": context, "text pair": hypothesis} NLI models normally output: 'entailment', 'neutral', or 'contradiction' label = result 'label' .lower In GraphEval, anything other than 'entailment' is flagged as a hallucination is hallucinated = label = 'entailment' return is hallucinated, label, hypothesis Running the evaluation pipeline evaluation results = print "\n--- GraphEval Results ---" for t in extracted triples: is hallucinated, nli label, hypothesis = evaluate triple source context, t evaluation results.append is hallucinated, nli label status = "🚨 HALLUCINATION" if is hallucinated else "✅ GROUNDED" print f"{status} | Triple: {t} | NLI Output: {nli label}" Output: --- GraphEval Results --- ✅ GROUNDED | Triple: 'GraphEval', 'is', 'evaluation framework' | NLI Output: entailment ✅ GROUNDED | Triple: 'GraphEval', 'uses', 'Knowledge Graphs' | NLI Output: entailment 🚨 HALLUCINATION | Triple: 'GraphEval', 'requires', 'expensive enterprise server farm' | NLI Output: neutral As we expected, the last triple in the knowledge graph is detected as a hallucination. To finish with a visual touch, we can also display the knowledge graph of the original LLM response alongside the detection results: python import networkx as nx import matplotlib.pyplot as plt import matplotlib.patches as mpatches def visualize grapheval triples, eval results : G = nx.DiGraph edge colors = for triple, res in zip triples, eval results : sub, rel, obj = triple is hallucinated = res 0 G.add node sub G.add node obj G.add edge sub, obj, label=rel Color-code the edges based on the NLI evaluation edge colors.append 'red' if is hallucinated else 'green' Set up the plot plt.figure figsize= 10, 6 pos = nx.spring layout G, seed=42 Draw nodes nx.draw networkx nodes G, pos, node color='lightblue', node size=2500 nx.draw networkx labels G, pos, font size=10, font weight='bold' Draw edges and labels nx.draw networkx edges G, pos, edge color=edge colors, width=2.5, arrowsize=20 edge labels = nx.get edge attributes G, 'label' nx.draw networkx edge labels G, pos, edge labels=edge labels, font color='black' Add legend green patch = mpatches.Patch color='green', label='Grounded Entailment ' red patch = mpatches.Patch color='red', label='Hallucination Neutral/Contradiction ' plt.legend handles= green patch, red patch , loc='lower right' plt.title "GraphEval Hallucination Map", fontsize=14, fontweight='bold' plt.axis 'off' plt.tight layout plt.show Render the knowledge graph visualize grapheval extracted triples, evaluation results Resulting visualization: Closing Remarks GraphEval is an evaluation methodology proposed to help detect and localize the root cause of hallucinations in LLM outputs. This article turned its key principles and methodological stages into a simulated practical scenario to better understand its usefulness and its key implications for potential implementation in production systems. is a leader, writer, speaker, and adviser in AI, machine learning, deep learning & LLMs. He trains and guides others in harnessing AI in the real world. Iván Palomares Carrascosa https://www.linkedin.com/in/ivanpc/