GraphRAG Evaluation: How to Test Retrieval Quality Before You Ship

FalkorDB published a GraphRAG evaluation methodology that measures retrieval quality before deployment, built around a 200-500 question test set spanning single-hop factual, multi-hop relational, temporal, and contradictory/ambiguous queries, each paired with a ground-truth answer and ground-truth subgraph. The framework instruments four retrieval-layer metrics, including entity recall, computed as the fraction of ground-truth entities present in the retrieved subgraph, with the company citing a baseline that scores below 0.85 on single-hop questions tend to indicate problems. FalkorDB's own analysis found the accuracy gap between graph and vector-only retrieval is smallest on single-fact lookups and largest on summarization, complex reasoning, and queries connecting information across multiple sources.

Most GraphRAG failures surface in production. A user asks a multi-hop question, the pipeline returns a confident answer, and the answer is wrong. The model didn’t hallucinate. Retrieval failed: the right entities were never fetched, the relationship path was broken, or the context window contained stale data. By the time you see that failure, it has already reached a user. Retrieval quality is measurable before you ship. Graph-based retrieval has structural properties that make it more testable than vector search alone: you can inspect the exact subgraph returned, verify entity presence, trace relationship paths, and check grounding against source nodes. You need a methodology that exercises those properties systematically. This post covers that methodology: how to build a test set that actually represents production load, which metrics to instrument, how to isolate retrieval failures from reasoning failures, and a starter harness you can adapt to your own pipeline. Benchmark context: our analysis of where graph retrieval diverges from vector-only approaches https://www.falkordb.com/reference/graphrag-vs-vector-rag-accuracy-gap/ shows the accuracy gap is smallest on single-fact lookups and largest on summarization, complex reasoning, and queries that require connecting information across multiple sources. Those are the query types this evaluation framework is built to stress-test. Build a Test Set That Reflects Production The most common mistake in RAG evaluation is testing with questions that are too easy. Simple factual lookups “What is X?” will pass almost any retrieval system. Production users ask harder things. A representative test set should include questions across four categories: - Single-hop factual: one entity, one attribute. These establish a baseline and catch basic indexing failures. - Multi-hop relational: two or more relationship traversals required. “Which engineers worked on projects that used the same vendor as Project Alpha?” These are where graph retrieval earns its keep. - Temporal: questions where the correct answer depends on a specific time window or on the ordering of events. “What was the team structure before the Q3 reorganization?” - Contradictory or ambiguous: questions where the graph contains conflicting information, either because source data is inconsistent or because the knowledge base has been partially updated. These test whether your pipeline surfaces uncertainty or silently picks one answer. Sourcing questions from production logs If you have a deployed system, mine your query logs. Cluster semantically similar questions, then sample from each cluster proportionally to its frequency. As a rough target, 200-500 questions gives you enough volume to detect meaningful metric differences between retrieval strategies; smaller sets tend to produce noisy comparisons, especially across four question categories. If you are pre-launch, generate questions programmatically from your graph schema. For each relationship type in your schema, write at least two questions that require traversing it. For each entity type with a time-indexed property, write at least one temporal question. Every question in your test set needs a ground-truth answer and a ground-truth subgraph. The subgraph is the set of nodes and edges a correct retrieval must return. Without it, you can only evaluate the final answer; you cannot diagnose where the pipeline broke. The Four Metrics That Matter Once you have a ground-truth test set, measure these four things at the retrieval layer, before the LLM ever sees the context. Entity recall The fraction of ground-truth entities present in the retrieved subgraph. entity recall = |retrieved entities ∩ ground truth entities| / |ground truth entities| These thresholds are starting points to calibrate against your own data, not universal standards. As a baseline: scores below 0.85 on single-hop questions tend to indicate an indexing or entity resolution problem. On multi-hop questions, scores below 0.70 often mean traversal depth is too shallow or relationship extraction during ingestion was lossy. Relationship recall Entity recall is necessary but not sufficient. You can retrieve all the right nodes while missing the edges that connect them, which breaks any reasoning that depends on path structure. relationship recall = |retrieved edges ∩ ground truth edges| / |ground truth edges| Track this separately from entity recall. A pipeline with high entity recall but low relationship recall is fetching islands of context, not connected subgraphs. Answer grounding After the LLM generates an answer, verify that every factual claim in the answer can be traced to a node or edge in the retrieved subgraph. This is the grounding score. Grounding is distinct from answer correctness. An answer can be factually correct but ungrounded the LLM used parametric knowledge rather than retrieved context , or it can be grounded but incorrect the retrieved context itself was wrong . You need both signals. Automate grounding checks by extracting named entities and claims from the LLM output and running them against the retrieved subgraph. Flag any claim that has no supporting node. Citation coverage For systems that surface source references, measure what fraction of cited sources actually appear in the retrieved subgraph. A citation that has no corresponding node in the retrieved context was invented by the model during generation, making it a signal of reasoning failure, not retrieval failure. Track it separately from retrieval metrics. | Metric | What it catches | Starting threshold | |---|---|---| | Entity recall | Missing nodes, shallow traversal | 0.85 single-hop , 0.70 multi-hop | | Relationship recall | Disconnected subgraphs, lossy ingestion | 0.80 | | Answer grounding | LLM bypassing retrieved context | 0.90 | | Citation coverage | Hallucinated references | 0.95 | Separate Retrieval Failures from Reasoning Failures When a GraphRAG pipeline returns a wrong answer, there are two possible culprits: the retrieval layer did not return the right context, or the LLM reasoned incorrectly over correct context. These require different fixes, and conflating them wastes debugging time. Use this diagnostic protocol: 1. Run retrieval in isolation. Execute the retrieval step and log the exact subgraph returned. Compare it against the ground-truth subgraph for that question. If entity recall or relationship recall is below threshold, the failure is retrieval-side. Stop here and fix ingestion, traversal depth, or query expansion. 2. Inject ground-truth context. For questions where retrieval passed but the final answer is still wrong, replace the retrieved context with the ground-truth subgraph and re-run the LLM. If the answer is now correct, you have confirmed a retrieval failure that your metrics missed. If the answer is still wrong with perfect context, the failure is in LLM reasoning or prompt construction. 3. Test temporal correctness explicitly. For time-indexed questions, verify that the retrieved subgraph reflects the correct time window. Store valid from and valid to timestamps on relationship edges and filter during retrieval. Then check whether the LLM’s answer respects the temporal constraint. In practice, temporal errors most often originate in retrieval rather than reasoning, but confirm this with the injection test before drawing conclusions. 4. Test contradiction handling. For questions where your graph contains conflicting nodes two sources asserting different values for the same attribute , check whether your pipeline surfaces the conflict or silently resolves it. A well-behaved system should either return both values with provenance or explicitly flag the inconsistency. A system that picks one silently is a liability. Key diagnostic principle: fix retrieval failures before tuning prompts. Prompt engineering on top of broken retrieval produces a system that sounds confident and is wrong. Fix the subgraph first. A Starter Evaluation Harness The following structure gives you a working baseline. Adapt the retrieval calls to your own pipeline interface. python from dataclasses import dataclass, field from typing import List, Literal, Set QuestionType = Literal "single hop", "multi hop", "temporal", "contradiction" Entity recall thresholds differ by question type. These are calibration starting points, not universal standards. ENTITY RECALL THRESHOLD = {"single hop": 0.85, "multi hop": 0.70, "temporal": 0.85, "contradiction": 0.70} REL RECALL THRESHOLD = 0.80 GROUNDING THRESHOLD and CITATION THRESHOLD are reference values for your own threshold checks inside pipeline.check grounding / check citations. GROUNDING THRESHOLD = 0.90 CITATION THRESHOLD = 0.95 @dataclass class TestCase: question: str question type: QuestionType ground truth answer: str ground truth key facts: List str facts a correct answer must contain ground truth entities: Set str ground truth edges: Set tuple source id, relation, target id ground truth subgraph: object the subgraph object for injection testing @dataclass class EvalResult: question: str question type: QuestionType entity recall: float relationship recall: float grounding score: float citation coverage: float answer correct: bool retrieval failure: bool missed retrieval failure: bool metrics passed but answer wrong until gt context injected reasoning failure: bool notes: List str = field default factory=list def check answer correctness answer: str, key facts: List str , judge - bool: """ Use a judge model to verify the answer covers the required key facts. Exact-string matching is too brittle for natural language answers; a judge prompt handles paraphrase and ordering differences. """ return judge.check answer, key facts def evaluate case tc: TestCase, pipeline, judge - EvalResult: Step 1: retrieval in isolation subgraph = pipeline.retrieve tc.question retrieved entities = set subgraph.node ids retrieved edges = set subgraph.edge tuples entity recall = len retrieved entities & tc.ground truth entities / max len tc.ground truth entities , 1 rel recall = len retrieved edges & tc.ground truth edges / max len tc.ground truth edges , 1 Step 2: generation with retrieved context answer, citations = pipeline.generate tc.question, subgraph grounding = pipeline.check grounding answer, subgraph Citation coverage: fraction of cited sources present in the retrieved subgraph. Citations absent from the subgraph were invented by the model generation failure . citation cov = pipeline.check citations citations, subgraph Step 3: answer correctness via judge model answer correct = check answer correctness answer, tc.ground truth key facts, judge Step 4: failure attribution er threshold = ENTITY RECALL THRESHOLD tc.question type retrieval fail = entity recall < er threshold or rel recall < REL RECALL THRESHOLD missed retrieval fail = False reasoning fail = False if not retrieval fail and not answer correct: Metrics looked fine but answer is wrong. Inject ground-truth context to isolate cause. gt answer, = pipeline.generate tc.question, tc.ground truth subgraph gt correct = check answer correctness gt answer, tc.ground truth key facts, judge if gt correct: Right with perfect context, wrong with retrieved context: retrieval was subtly wrong in a way the recall metrics didn't catch. missed retrieval fail = True else: Still wrong with perfect context: the problem is in reasoning or prompt construction. reasoning fail = True return EvalResult question=tc.question, question type=tc.question type, entity recall=entity recall, relationship recall=rel recall, grounding score=grounding, citation coverage=citation cov, answer correct=answer correct, retrieval failure=retrieval fail, missed retrieval failure=missed retrieval fail, reasoning failure=reasoning fail, def run suite test cases: List TestCase , pipeline, judge - dict: if not test cases: return {} results = evaluate case tc, pipeline, judge for tc in test cases by type: dict = {} for r in results: by type.setdefault r.question type, .append r n = len results summary = { "overall": { "entity recall avg": sum r.entity recall for r in results / n, "relationship recall avg": sum r.relationship recall for r in results / n, "grounding avg": sum r.grounding score for r in results / n, "answer accuracy": sum r.answer correct for r in results / n, "retrieval failure rate": sum r.retrieval failure for r in results / n, "missed retrieval failure rate": sum r.missed retrieval failure for r in results / n, "reasoning failure rate": sum r.reasoning failure for r in results / n, } } for qtype, group in by type.items : g = len group summary qtype = { "entity recall avg": sum r.entity recall for r in group / g, "answer accuracy": sum r.answer correct for r in group / g, "retrieval failure rate": sum r.retrieval failure for r in group / g, "missed retrieval failure rate": sum r.missed retrieval failure for r in group / g, } return summary Pulling the retrieved subgraph from FalkorDB The harness above compares subgraph.node ids and subgraph.edge tuples against ground truth. If you are running on FalkorDB, pull those directly with Cypher: // Edges among retrieved nodes only. // Both ends must be in the retrieved set; otherwise the query returns edges // to nodes that were never retrieved, inflating relationship recall. MATCH n - r - m WHERE n.id IN $retrieved node ids AND m.id IN $retrieved node ids RETURN n.id AS source, type r AS relation, m.id AS target, r.valid from AS valid from, r.valid to AS valid to This returns the edges that exist among the retrieved nodes, which is an approximation of what the retriever actually returned the retriever may have fetched additional context not captured here . Compare the source, relation, target triples against tc.ground truth edges to compute relationship recall, and inspect valid from / valid to inline during the temporal correctness check. Running a fair comparison across retrieval strategies Run the harness against vector-only, graph-only, and hybrid configurations using the same test set. For the comparison to be valid, hold everything else constant: same LLM, same judge model, same source corpus, same chunking. The only variable that changes is the retrieval strategy. Report results by question type , not as a single aggregate. A hybrid strategy can perform better on multi-hop and temporal questions while vector-only performs comparably on single-hop factual lookups. Aggregating across types hides that signal and leads to bad architectural decisions. Our benchmark follows the same principle: same model and judge for every system, with results broken down by category. That is why the per-category numbers in that analysis https://www.falkordb.com/reference/graphrag-vs-vector-rag-accuracy-gap/ are more useful than the headline figures. Make Evaluation Part of the Build Cycle Treat this as a regression suite, not a pre-launch checklist. Run it every time you change ingestion logic, update your schema, or swap retrieval strategies. The test set you build from production questions compounds in value over time as it accumulates edge cases that real users surface. What you have after working through this: 1. A structured test set covering single-hop, multi-hop, temporal, and contradiction cases 2. Four retrieval-layer metrics measured before the LLM ever sees the context 3. A diagnostic protocol that separates retrieval failures from reasoning failures 4. A harness that runs the same evaluation across retrieval strategies with per-category reporting A perfect score on an aggregate benchmark tells you very little. Knowing which question types fail, and whether the failure is in retrieval or reasoning, tells you exactly where to spend engineering time.