Most GraphRAG failures surface in production. A user asks a multi-hop question, the pipeline returns a confident answer, and the answer is wrong. The model didn’t hallucinate. Retrieval failed: the right entities were never fetched, the relationship path was broken, or the context window contained stale data.
By the time you see that failure, it has already reached a user.
Retrieval quality is measurable before you ship. Graph-based retrieval has structural properties that make it more testable than vector search alone: you can inspect the exact subgraph returned, verify entity presence, trace relationship paths, and check grounding against source nodes. You need a methodology that exercises those properties systematically.
This post covers that methodology: how to build a test set that actually represents production load, which metrics to instrument, how to isolate retrieval failures from reasoning failures, and a starter harness you can adapt to your own pipeline.
Benchmark context: our analysis of where graph retrieval diverges from vector-only approaches shows the accuracy gap is smallest on single-fact lookups and largest on summarization, complex reasoning, and queries that require connecting information across multiple sources. Those are the query types this evaluation framework is built to stress-test.
Build a Test Set That Reflects Production #
The most common mistake in RAG evaluation is testing with questions that are too easy. Simple factual lookups (“What is X?”) will pass almost any retrieval system. Production users ask harder things.
A representative test set should include questions across four categories:
- Single-hop factual: one entity, one attribute. These establish a baseline and catch basic indexing failures.
- Multi-hop relational: two or more relationship traversals required. (“Which engineers worked on projects that used the same vendor as Project Alpha?”) These are where graph retrieval earns its keep.
- Temporal: questions where the correct answer depends on a specific time window or on the ordering of events. (“What was the team structure before the Q3 reorganization?”)
- Contradictory or ambiguous: questions where the graph contains conflicting information, either because source data is inconsistent or because the knowledge base has been partially updated. These test whether your pipeline surfaces uncertainty or silently picks one answer.
Sourcing questions from production logs
If you have a deployed system, mine your query logs. Cluster semantically similar questions, then sample from each cluster proportionally to its frequency. As a rough target, 200-500 questions gives you enough volume to detect meaningful metric differences between retrieval strategies; smaller sets tend to produce noisy comparisons, especially across four question categories.
If you are pre-launch, generate questions programmatically from your graph schema. For each relationship type in your schema, write at least two questions that require traversing it. For each entity type with a time-indexed property, write at least one temporal question.
Every question in your test set needs a ground-truth answer and a ground-truth subgraph. The subgraph is the set of nodes and edges a correct retrieval must return. Without it, you can only evaluate the final answer; you cannot diagnose where the pipeline broke.
The Four Metrics That Matter #
Once you have a ground-truth test set, measure these four things at the retrieval layer, before the LLM ever sees the context.
Entity recall
The fraction of ground-truth entities present in the retrieved subgraph.
entity_recall = |retrieved_entities ∩ ground_truth_entities| / |ground_truth_entities|
These thresholds are starting points to calibrate against your own data, not universal standards. As a baseline: scores below 0.85 on single-hop questions tend to indicate an indexing or entity resolution problem. On multi-hop questions, scores below 0.70 often mean traversal depth is too shallow or relationship extraction during ingestion was lossy.
Relationship recall
Entity recall is necessary but not sufficient. You can retrieve all the right nodes while missing the edges that connect them, which breaks any reasoning that depends on path structure.
relationship_recall = |retrieved_edges ∩ ground_truth_edges| / |ground_truth_edges|
Track this separately from entity recall. A pipeline with high entity recall but low relationship recall is fetching islands of context, not connected subgraphs.
Answer grounding
After the LLM generates an answer, verify that every factual claim in the answer can be traced to a node or edge in the retrieved subgraph. This is the grounding score.
Grounding is distinct from answer correctness. An answer can be factually correct but ungrounded (the LLM used parametric knowledge rather than retrieved context), or it can be grounded but incorrect (the retrieved context itself was wrong). You need both signals.
Automate grounding checks by extracting named entities and claims from the LLM output and running them against the retrieved subgraph. Flag any claim that has no supporting node.
Citation coverage
For systems that surface source references, measure what fraction of cited sources actually appear in the retrieved subgraph. A citation that has no corresponding node in the retrieved context was invented by the model during generation, making it a signal of reasoning failure, not retrieval failure. Track it separately from retrieval metrics.
| Metric | What it catches | Starting threshold |
|---|---|---|
| Entity recall | Missing nodes, shallow traversal | > 0.85 (single-hop), > 0.70 (multi-hop) |
| Relationship recall | Disconnected subgraphs, lossy ingestion | > 0.80 |
| Answer grounding | LLM bypassing retrieved context | > 0.90 |
| Citation coverage | Hallucinated references | > 0.95 |
Separate Retrieval Failures from Reasoning Failures #
When a GraphRAG pipeline returns a wrong answer, there are two possible culprits: the retrieval layer did not return the right context, or the LLM reasoned incorrectly over correct context. These require different fixes, and conflating them wastes debugging time.
Use this diagnostic protocol:
- Run retrieval in isolation. Execute the retrieval step and log the exact subgraph returned. Compare it against the ground-truth subgraph for that question. If entity recall or relationship recall is below threshold, the failure is retrieval-side. Stop here and fix ingestion, traversal depth, or query expansion.
- Inject ground-truth context. For questions where retrieval passed but the final answer is still wrong, replace the retrieved context with the ground-truth subgraph and re-run the LLM. If the answer is now correct, you have confirmed a retrieval failure that your metrics missed. If the answer is still wrong with perfect context, the failure is in LLM reasoning or prompt construction.
- Test temporal correctness explicitly. For time-indexed questions, verify that the retrieved subgraph reflects the correct time window. Store
valid_fromandvalid_totimestamps on relationship edges and filter during retrieval. Then check whether the LLM’s answer respects the temporal constraint. In practice, temporal errors most often originate in retrieval rather than reasoning, but confirm this with the injection test before drawing conclusions. - Test contradiction handling. For questions where your graph contains conflicting nodes (two sources asserting different values for the same attribute), check whether your pipeline surfaces the conflict or silently resolves it. A well-behaved system should either return both values with provenance or explicitly flag the inconsistency. A system that picks one silently is a liability.
Key diagnostic principle: fix retrieval failures before tuning prompts. Prompt engineering on top of broken retrieval produces a system that sounds confident and is wrong. Fix the subgraph first.
A Starter Evaluation Harness #
The following structure gives you a working baseline. Adapt the retrieval calls to your own pipeline interface.
from dataclasses import dataclass, field
from typing import List, Literal, Set
QuestionType = Literal["single_hop", "multi_hop", "temporal", "contradiction"]
ENTITY_RECALL_THRESHOLD = {"single_hop": 0.85, "multi_hop": 0.70, "temporal": 0.85, "contradiction": 0.70}
REL_RECALL_THRESHOLD = 0.80
GROUNDING_THRESHOLD = 0.90
CITATION_THRESHOLD = 0.95
@dataclass
class TestCase:
question: str
question_type: QuestionType
ground_truth_answer: str
ground_truth_key_facts: List[str] # facts a correct answer must contain
ground_truth_entities: Set[str]
ground_truth_edges: Set[tuple] # (source_id, relation, target_id)
ground_truth_subgraph: object # the subgraph object for injection testing
@dataclass
class EvalResult:
question: str
question_type: QuestionType
entity_recall: float
relationship_recall: float
grounding_score: float
citation_coverage: float
answer_correct: bool
retrieval_failure: bool
missed_retrieval_failure: bool # metrics passed but answer wrong until gt context injected
reasoning_failure: bool
notes: List[str] = field(default_factory=list)
def check_answer_correctness(answer: str, key_facts: List[str], judge) -> bool:
"""
Use a judge model to verify the answer covers the required key facts.
Exact-string matching is too brittle for natural language answers;
a judge prompt handles paraphrase and ordering differences.
"""
return judge.check(answer, key_facts)
def evaluate_case(tc: TestCase, pipeline, judge) -> EvalResult:
subgraph = pipeline.retrieve(tc.question)
retrieved_entities = set(subgraph.node_ids())
retrieved_edges = set(subgraph.edge_tuples())
entity_recall = len(retrieved_entities & tc.ground_truth_entities) / max(len(tc.ground_truth_entities), 1)
rel_recall = len(retrieved_edges & tc.ground_truth_edges) / max(len(tc.ground_truth_edges), 1)
answer, citations = pipeline.generate(tc.question, subgraph)
grounding = pipeline.check_grounding(answer, subgraph)
citation_cov = pipeline.check_citations(citations, subgraph)
answer_correct = check_answer_correctness(answer, tc.ground_truth_key_facts, judge)
er_threshold = ENTITY_RECALL_THRESHOLD[tc.question_type]
retrieval_fail = entity_recall < er_threshold or rel_recall < REL_RECALL_THRESHOLD
missed_retrieval_fail = False
reasoning_fail = False
if not retrieval_fail and not answer_correct:
gt_answer, _ = pipeline.generate(tc.question, tc.ground_truth_subgraph)
gt_correct = check_answer_correctness(gt_answer, tc.ground_truth_key_facts, judge)
if gt_correct:
missed_retrieval_fail = True
else:
reasoning_fail = True
return EvalResult(
question=tc.question,
question_type=tc.question_type,
entity_recall=entity_recall,
relationship_recall=rel_recall,
grounding_score=grounding,
citation_coverage=citation_cov,
answer_correct=answer_correct,
retrieval_failure=retrieval_fail,
missed_retrieval_failure=missed_retrieval_fail,
reasoning_failure=reasoning_fail,
)
def run_suite(test_cases: List[TestCase], pipeline, judge) -> dict:
if not test_cases:
return {}
results = [evaluate_case(tc, pipeline, judge) for tc in test_cases]
by_type: dict = {}
for r in results:
by_type.setdefault(r.question_type, []).append(r)
n = len(results)
summary = {
"overall": {
"entity_recall_avg": sum(r.entity_recall for r in results) / n,
"relationship_recall_avg": sum(r.relationship_recall for r in results) / n,
"grounding_avg": sum(r.grounding_score for r in results) / n,
"answer_accuracy": sum(r.answer_correct for r in results) / n,
"retrieval_failure_rate": sum(r.retrieval_failure for r in results) / n,
"missed_retrieval_failure_rate": sum(r.missed_retrieval_failure for r in results) / n,
"reasoning_failure_rate": sum(r.reasoning_failure for r in results) / n,
}
}
for qtype, group in by_type.items():
g = len(group)
summary[qtype] = {
"entity_recall_avg": sum(r.entity_recall for r in group) / g,
"answer_accuracy": sum(r.answer_correct for r in group) / g,
"retrieval_failure_rate": sum(r.retrieval_failure for r in group) / g,
"missed_retrieval_failure_rate": sum(r.missed_retrieval_failure for r in group) / g,
}
return summary
Pulling the retrieved subgraph from FalkorDB
The harness above compares subgraph.node_ids() and subgraph.edge_tuples() against ground truth. If you are running on FalkorDB, pull those directly with Cypher:
// Edges among retrieved nodes only.
// Both ends must be in the retrieved set; otherwise the query returns edges
// to nodes that were never retrieved, inflating relationship recall.
MATCH (n)-[r]->(m)
WHERE n.id IN $retrieved_node_ids AND m.id IN $retrieved_node_ids
RETURN n.id AS source, type(r) AS relation, m.id AS target,
r.valid_from AS valid_from, r.valid_to AS valid_to
This returns the edges that exist among the retrieved nodes, which is an approximation of what the retriever actually returned (the retriever may have fetched additional context not captured here). Compare the (source, relation, target) triples against tc.ground_truth_edges to compute relationship recall, and inspect valid_from / valid_to inline during the temporal correctness check.
Running a fair comparison across retrieval strategies
Run the harness against vector-only, graph-only, and hybrid configurations using the same test set. For the comparison to be valid, hold everything else constant: same LLM, same judge model, same source corpus, same chunking. The only variable that changes is the retrieval strategy.
Report results by question_type, not as a single aggregate. A hybrid strategy can perform better on multi-hop and temporal questions while vector-only performs comparably on single-hop factual lookups. Aggregating across types hides that signal and leads to bad architectural decisions. Our benchmark follows the same principle: same model and judge for every system, with results broken down by category. That is why the per-category numbers in that analysis are more useful than the headline figures.
Make Evaluation Part of the Build Cycle #
Treat this as a regression suite, not a pre-launch checklist. Run it every time you change ingestion logic, update your schema, or swap retrieval strategies. The test set you build from production questions compounds in value over time as it accumulates edge cases that real users surface.
What you have after working through this:
- A structured test set covering single-hop, multi-hop, temporal, and contradiction cases
- Four retrieval-layer metrics measured before the LLM ever sees the context
- A diagnostic protocol that separates retrieval failures from reasoning failures
- A harness that runs the same evaluation across retrieval strategies with per-category reporting
A perfect score on an aggregate benchmark tells you very little. Knowing which question types fail, and whether the failure is in retrieval or reasoning, tells you exactly where to spend engineering time.