Knowledge graphs vs RAG: when graphs actually win

Context Engineering Structured Context

When to process AI context: ingestion, dreaming, query

Key takeaway

Knowledge graphs vs RAG is not an either/or. Graph-based retrieval wins on multi-hop reasoning and global summarization: Microsoft's GraphRAG beat vector RAG on answer comprehensiveness 72 to 83 percent of the time, and HippoRAG improved multi-hop recall by up to 20 percent. But standard vector RAG matches or beats graphs on simple fact lookups while costing far less to index. The right choice follows from query complexity, not novelty, and the most reliable systems layer both.

The pitch for knowledge graphs sounds a lot like the pitch for long context windows did a year ago. Vector search is crude, the argument goes. It chops documents into chunks and matches them by similarity, throwing away how the facts actually relate. Knowledge graphs keep those relationships, so graph retrieval will replace plain RAG. The benchmarks published through 2026 tell a more specific story. Knowledge graphs vs RAG is not a contest one technique wins outright. It is a question of what your queries actually need, and the answer flips depending on whether a question lives in one passage or spans many.

This matters because the two approaches have very different cost structures. Getting the choice wrong means either paying for graph infrastructure that does nothing for your query mix, or running vector search on questions it structurally cannot answer. The data now exists to make the call deliberately.

Graph-based retrieval beats vector RAG on questions that span multiple documents and loses to it on simple fact lookups. The clearest evidence comes from GraphRAG-Bench (arXiv 2506.05690), which sorts queries into four levels of difficulty: fact retrieval, complex reasoning, contextual summarization, and creative generation. The results split cleanly along that axis.

On simple fact retrieval, basic RAG scored 60.92 percent accuracy while graph methods landed lower, between 49.29 and 60.14 percent. The paper’s own summary is blunt: “basic RAG is comparable to or outperforms GraphRAG in simple fact retrieval tasks,” because graph structures introduce “redundant or noisy information for simpler queries.” On complex reasoning, the order reverses. HippoRAG reached 53.38 percent accuracy against vector RAG’s 42.93 percent, and the gap on evidence recall was wider still: graph methods recalled 87.9 to 90.9 percent of the needed evidence versus 64.47 percent for vector retrieval.

The takeaway is not that one method is better. It is that the right method is a function of query complexity, and a real workload contains both kinds of question.

Knowledge graphs win decisively on multi-hop reasoning and whole-corpus synthesis, the two places vector retrieval is structurally weakest. A multi-hop question chains several facts that live in different documents: “which dependencies of the billing service were changed by the engineer who owns the auth migration” requires three linked lookups, not one. Vector search pulls passages that each look relevant but rarely returns the path between them.

HippoRAG (arXiv 2405.14831), presented at NeurIPS 2024, demonstrated the gap. It improved multi-hop question answering by up to 20 percent over strong baselines on benchmarks like 2WikiMultiHopQA, and its single-step graph retrieval matched iterative retrieval methods like IRCoT while running 6 to 13 times faster. The relationships were already encoded in the graph, so the system did not need repeated retrieval round-trips to assemble a chain of evidence.

Global questions show the same pattern. Microsoft’s GraphRAG paper (arXiv 2404.16130) tested “global sensemaking” queries over corpora in the million-token range, the kind of question that asks what a whole dataset is about rather than retrieving one fact from it. Graph-based retrieval beat a vector RAG baseline on answer comprehensiveness 72 to 83 percent of the time and on diversity 62 to 82 percent of the time, all at p < .001. It also did so efficiently: hierarchical community summaries built from the graph used up to 97 percent fewer tokens per query than summarizing the source text directly. When a question is about the structure of a corpus, encoding that structure in the index pays off, both in quality and in tokens. This is the same lesson as structured context beating raw text: the shape you give information before retrieval determines what the model can do with it.

Vector RAG remains the better default for specific fact retrieval and for any system where indexing cost matters, which is most of them. The accuracy case was already covered: on single-fact questions, the answer sits in one passage, and graph traversal only adds entities and relationships that crowd the context window without adding signal. The reason “RAG is not enough” sometimes is naive retrieval, not vector search itself, and naive retrieval is cheap to fix relative to standing up a graph.

The cost gap is the more decisive factor. Building a vector index means embedding chunks, a fast and well-understood operation. Building a knowledge graph usually means running an LLM over every chunk to extract entities and typed relationships, which is dramatically more expensive. Microsoft’s own LazyGraphRAG work makes the scale of this concrete: it reaches comparable answer quality at roughly 0.1 percent of full GraphRAG’s indexing cost. A method whose cost can be cut by three orders of magnitude was paying a steep tax to begin with.

Query-time cost can blow up too. GraphRAG-Bench measured global graph retrieval reaching up to 40,000 tokens of context for a single query, against roughly 879 tokens for vanilla RAG. That 40x inflation does not just cost money. Past a point it degrades accuracy, because more loosely related context is exactly what drives context rot.

Maintenance is the quieter cost. A vector index updates by re-embedding the documents that changed, an operation scoped to exactly what moved. A knowledge graph has to reconcile new entities and relationships against existing ones, which can ripple through the regions of the graph they touch. For a corpus that changes daily, that reconciliation work recurs forever, and it is the kind of overhead that looks small in a demo and dominates the bill in production.

The decision comes down to a handful of dimensions, and laying them side by side makes the tradeoff legible.

No row makes graphs or vectors the universal winner. Each is a different bet about which class of question dominates your traffic.

Choosing between graph and vector retrieval is a context engineering decision, not a database procurement one. Both methods answer the same underlying question: what does the model actually need to see to answer this query, and in what structure. Vector search delivers the most similar passages. Graph traversal delivers the connected ones. The failure mode in each case is a context failure, either missing the relationships or drowning the answer in them.

That framing also dissolves the “which one won” debate. The strongest 2026 systems treat graph and vector retrieval as two layers of the same stack rather than rival products. HippoRAG 2, for example, uses vector similarity to find entry points into the graph and traversal to follow relationships from there, beating either method alone on complex reasoning while staying close to vector RAG on simple lookups. The same logic that makes hybrid retrieval the winning pattern for long context vs RAG applies here: combine the methods so each handles the query class it is built for.

The practical version of this is to capture relationships once, at write time, so an agent does not have to reconstruct them on every query. An agent connected to a Wire container experiences this directly: relationships between entries are resolved when content is written, so calling wire_navigate

walks from an entry to its related entries without a separate graph-build pass or repeated retrieval round-trips (how Wire’s retrieval is benchmarked). The relationships are part of the index, not something the model infers from scattered chunks.

Start with your query mix, not the technology. If most questions are “find the passage that states X,” vector RAG is the right default, and the effort is better spent on chunking, ranking, and prompt construction than on graph infrastructure. If a meaningful share of questions are “connect facts across documents” or “summarize what this whole corpus says,” graph structure earns its cost, and a hybrid that keeps vector search for the easy questions will usually beat going all-in on either.

Then weigh the budget honestly. Graph indexing is expensive and the tooling moves fast, so a team without multi-hop or global queries in its workload is paying for capability it will not use. A team drowning in multi-hop failures is leaving accuracy on the table by forcing every question through similarity search. The benchmarks give you the numbers to make that call: a roughly 10-point accuracy swing in each direction depending on query type, against an indexing cost that can differ by 1000x.

The “knowledge graphs replace RAG” narrative is the same shape as “long context killed RAG,” and it resolves the same way. Neither technique wins in general. The tradeoffs are quantifiable, they break cleanly along query complexity, and the systems that perform best in production use both, each where it is strongest.

Sources: From Local to Global: A Graph RAG Approach to Query-Focused Summarization (arXiv 2404.16130) · When to use Graphs in RAG / GraphRAG-Bench (arXiv 2506.05690) · HippoRAG: Neurobiologically Inspired Long-Term Memory for LLMs (arXiv 2405.14831) · Microsoft Research: LazyGraphRAG Wire transforms your documents into structured, AI-optimized context containers. Upload files, get MCP tools instantly.