cd /news/large-language-models/the-same-graphrag-comparison-wins-an… · home topics large-language-models article
[ARTICLE · art-115289] src=dev.to ↗ pub= topic=large-language-models verified=true sentiment=· neutral

The Same GraphRAG Comparison Wins and Loses. It Depends Which Instrument Judged It.

A developer's analysis of GraphRAG benchmark comparisons reveals that results vary dramatically depending on whether an LLM judge or ground-truth metrics are used. The same paper reports GraphRAG winning on LLM-judged comprehensiveness and diversity but losing on ROUGE-2 against gold answers, highlighting the importance of understanding evaluation methodology before citing benchmark wins.

read2 min views1 publishedAug 29, 2026

One finding, and it is not about graphs. It is about how the result you are quoting was produced.

For breadth-oriented sensemaking questions, GraphRAG community summaries reached 72 to 83 percent comprehensiveness and 62 to 82 percent diversity win rates, and root-level summaries cut answer tokens by about 97 percent (arXiv:2404.16130). Those are the numbers the category quotes. They were produced by an LLM judge, with no gold answers.

Score the same kind of comparison against ground truth and it turns over. On ROUGE-2 against gold answers, GraphRAG loses to plain RAG: 6.99 against 10.08 on SQuALITY, and 3.23 against 6.32 on QMSum. The same paper reports that summary-ordering position bias can flip the LLM judge's preference on its own (arXiv:2502.11371).

That is the part worth sitting with. Neither number is wrong and neither paper is sloppy.

One measures which answer a model prefers. The other measures overlap with a gold answer. They are different questions, so they are allowed to have different answers, and a comparison that looks decisive is often just one of the two questions asked loudly.

The same seam runs through the retrieval verdicts. On GraphRAG-Bench, "RAG (w rerank)" beat "GraphRAG (local)" 60.92 to 49.29 on fact retrieval, while "HippoRAG2" beat "RAG (w rerank)" 53.38 to 42.93 on complex reasoning (arXiv:2506.05690). One table, two verdicts, depending on which row of it you were looking for.

"Graph equals expensive" is too crude to act on. On the Novel split of the same benchmark, query cost spans more than two orders of magnitude within the graph methods themselves: about 331,375 total query tokens for "MS-GraphRAG(global)", about 100,832 for LightRAG, about 4,204 for Fast-GraphRAG, about 1,008 for "HippoRAG2", against about 879 for vanilla RAG.

Index build shows the same spread: roughly 9.2 million tokens for "HippoRAG2" against roughly 115.5 million for GraphRAG, about twelvefold (arXiv:2502.14802, Table 12).

Those two figures come from different primaries with different method sets, and no single primary puts index-build cost and query cost for the same method on one axis. Which is itself the lesson: the tidy cost model you want does not exist yet, and assembling one from two papers gives you a useful heuristic, not a controlled comparison.

Before accepting any retrieval win, mine included:

Was this judged by a model, or scored against ground truth?

It takes ten seconds to check and it changes what the number means. A model-judged win says an evaluator preferred that answer. A ground-truth score says the answer overlapped with a known correct one. Quoting the first while implying the second is how a benchmark verdict gets mistaken for a decision framework.

Disclosure: I work on Mnemoverse, a memory engine for AI agents. Every number above comes from a linked paper, none of them ours. The longer version, with the full decision matrix, is on the original page.

── more in #large-language-models 4 stories · sorted by recency
── more on @graphrag 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
Live at https://your-agent.zahid.host
Get free account → Pricing
from €0/mo · no card required
LIVE [news/the-same-graphrag-co…] indexed:0 read:2min 2026-08-29 ·