# Does RAG Need Better Retrieval — or Better Relationships? > Source: > Published: 2026-09-17 19:17:28+00:00 I started this experiment for a slightly different reason. I was playing with a system that could reconstruct context across documents. Not summarize documents. Not retrieve the most similar chunks. Reconstruct context. For example, suppose I have two documents: In 2022, the team chose PostgreSQL. And somewhere else: After the migration problems, the team moved to DynamoDB in 2024. Neither document contains the answer to: Why did the database decision change? The answer lives in the relationship between them. That got me thinking about something else. If a system can reliably recover these relationships, could it also use them to create structured training data? Before going there, I wanted to know whether the relationships actually mattered. So I built a small experiment. And the first result was much stronger than I expected. The obvious problem with testing relationships in RAG is retrieval. If the experiment gets better answers after adding a graph, what actually improved? Maybe the graph found documents that retrieval missed. Maybe it simply retrieved more useful evidence. Maybe the model got more context. Maybe the graph structure itself helped. Those are very different explanations. I wanted to isolate one of them. So I made the retrieved documents identical in both conditions. No re-ranking. No additional documents. No better retrieval. The model gets the same documents. The only difference is whether it also receives explicit relationships between those documents. That became the experiment: **Does explicitly stated relational context improve reasoning when retrieval is held constant?** I built a synthetic corpus of 254 business documents. The documents cover things like billing, DevOps and HR. Then I created 60 multi-hop cases with manually defined ground truth: Some cases require reasoning across several documents. Six of the supersession cases involve four to six documents with unrelated distractors. I deliberately made some of the relationships hard to recover from individual documents. For example, two documents can both be perfectly valid while representing different states of the same decision. The important information isn't inside either document. It's in: `A → superseded by → B` or: `A ↔ contradicts ↔ B` `A → belongs to → cluster B` There is also a meaningful negative case: `A → no known relation → B` That one turned out to matter more than I expected. I used a lexical BM25-style top-5 retrieval over the full corpus. It isn't a production BM25 implementation. It's a deliberately simple lexical baseline. Mean recall of the oracle documents in the top five was 0.67. 27 of the 60 cases had full recall. 17 had partial recall. 10 had low recall. Six had zero recall, all of them negative cases. This is a real limitation. If retrieval misses evidence, a relational hint can potentially compensate for that weakness. So I don't want to pretend this experiment answers what happens with a very strong production retrieval system. But it gives us something useful: C and D see **byte-identical retrieved documents**. That part is controlled. The baseline looks roughly like this: ``` retrieved documents ↓ LLM ↓ answer ``` The experimental condition adds a small structured block: ``` retrieved documents + explicit relationships ↓ LLM ↓ answer ``` For example: ``` kind: superseded from: database-choice-2022.md to: database-choice-2024.md reason: migration failure ``` The model was told that the knowledge-base engine had detected the relation and that it should use the hint if it was consistent with the documents. Nothing else changed. I ran the main experiment with two models. | Model | No relation | Typed relations | |---|---|---| | DeepSeek | 77% | 94% | | GPT-4o-mini | 77% | 98% | The gain was: **+17 percentage points** for DeepSeek. **+21 points** for GPT-4o-mini. That was enough to make me stop and ask a more annoying question: **What exactly did I measure?** Because “adding a graph improves RAG” would be a very easy conclusion. And probably a premature one. The relation hints were only around 17 words. Perhaps the model simply benefited from getting more information. So I added a control with approximately the same amount of unrelated text. Parking. Cafeteria. Meeting rooms. No relation labels. No document names. No similarity scores. That condition scored 53%. The baseline was 77%. The relational condition was 98%. So “more tokens” doesn't explain the result. At least not by itself. The relational information was presented as fields: ``` kind: from: to: reason: ``` Maybe that formatting alone makes the model perform better. So I took the same semantic information and put it into ordinary prose. Something like: Note: B reflects the current state of this decision. A is the previous version. Same information. No structured relation fields. That scored 53%. The structured version scored 98%. This result bothered me. Because now I had learned something I hadn't originally set out to test: **how the relationship is presented may matter almost as much as the relationship itself.** And I had a new confound. Maybe the model doesn't actually reason over the relationship. Maybe it just trusts the block because I told it that a “knowledge base engine” generated it. I'll come back to that. This was the test I cared about most. If relationships really carry useful information, incorrect relationships should matter. I corrupted the identifying fields of the relation hints. At 20% corruption, accuracy dropped from 98% to 53%. At 40%, it was 52%. That is a dramatic failure. But there is an important detail here. I did **not** test every corruption level. I don't know what happens at 5%. I don't know what happens at 10%. I don't know whether another corruption mechanism would behave differently. So I can't say: “80% correct relations are useless.” The experiment doesn't show that. It shows something narrower: Under this particular corruption mechanism, introducing 20% incorrect relations almost completely destroyed the observed benefit. That distinction matters. This was probably the most interesting part of the experiment. If relationships help because they connect documents, perhaps we don't need explicit relation types. Maybe a similarity graph is enough. So I created another condition where the model received document-to-document similarity edges instead of typed relations. The result improved substantially: | Model | No hint | Similarity | Typed relation | |---|---|---|---| | DeepSeek | 76% | 88% | 96% | | GPT-4o-mini | 76% | 93% | 98% | Similarity clearly helped. But it didn't behave like typed relations. There was a particularly interesting case. For DeepSeek, similarity actually beat typed relations on the link cases: **79% vs 71%.** The likely explanation is still only a hypothesis. The typed hint contained a `reason` field that may have distracted the model. The negative cases were different. Similarity hurt them. The model saw related-looking documents and tended to assume that a relationship existed. The typed relation could explicitly say: ``` kind: none ``` Similarity says: These things look related. A typed relation can say: These things are related in this specific way. Or: These things are not related. Those are different pieces of information. I added three more controls: **Shuffled relations** Give each case a relation from another case of the same kind. **Token-matched text** Give the model unrelated text with approximately the same length as the relation hint. **In-document prose** Put the same semantic relation into ordinary document text. All three landed around 52–53%. The structured, correct relation condition was 98%. The shuffled condition also passed an anti-leakage check: none of the 60 shuffled hints happened to be correct. At this point I was reasonably confident that the result wasn't just: “Give the LLM some extra words that look vaguely useful.” But I still wasn't confident about *why* the structured relation worked. There is one explanation my controls don't eliminate. Maybe the model isn't using the relational structure as relational structure. Maybe it sees: `Retineo relation: kind=superseded ...` and interprets that as privileged metadata. In other words: “The system says this is the answer, so I'll trust it.” That would produce a large improvement without requiring the model to reconstruct anything from the relationship itself. The in-document prose control actually makes this possibility more interesting. The two conditions contain the same semantic information. Yet the structured version was dramatically more effective. So I can't currently separate: **relational reasoning** from **authority/salience effects caused by the representation.** That's the next control I want to run. My retrieval baseline is weak. That's not an incidental detail. Mean top-5 recall was only 0.67, so there is plenty of room for a relational hint to compensate for retrieval failures. Imagine that the model needs documents A, B and C. The retriever returns A and B. Then I give it a relation saying: ``` A → related_to → C ``` I haven't really solved the retrieval problem. I've given the model a shortcut around it. So the next experiment is becoming pretty obvious. I want to vary retrieval depth while keeping the relational condition intact. ``` shallow retrieval × no relation / relation ``` then: ``` deeper retrieval × no relation / relation ``` The question is not whether deeper retrieval improves the baseline. Of course it can. The interesting question is what happens to the **gap** between the two conditions. If: ``` +21 pp ↓ +10 pp ↓ +2 pp ``` then the relational hint may mostly be compensating for weak retrieval. If the gap remains large as retrieval quality improves, the story gets more interesting. But this experiment needs some care. Simply dumping 100 documents into the context changes the amount of text the model has to process, so that would introduce another variable. I want to separate **more available evidence** from **more text in the prompt**. Here is the version I'm comfortable with. On this synthetic corpus, with this retrieval setup and these two models: **Correct typed cross-document relations improved accuracy when the retrieved documents were identical.** The effect survived controls for extra tokens, shuffled relations, and expressing the same semantic information as ordinary document prose. Similarity structure also helped, but typed relations provided additional information in several cases, particularly where relation type or explicit absence mattered. That's the observation. The explanation is still open. I don't know yet how much of the effect comes from: And I definitely don't know whether the effect survives real organizational data. The corpus is synthetic. The retrieval is simplified. There are only two models. Those are real boundaries on the result, not footnotes to hide at the end. There was a practical reason I ran the experiment in the first place. I was thinking about whether a context-reconstruction system could help create structured datasets. Take a collection of documents. Find candidate relationships. Classify them: ``` supersedes contradicts belongs_to references none ``` Then use those relationships as structured training data. That sounds useful. But it also creates a nasty circular problem. If I use the system to generate relational labels, and then use those labels to demonstrate that relational context helps models, I may just be measuring my own assumptions. So I stopped. Before thinking about dataset generation, I wanted to know whether the representation itself had a measurable effect. That led to this experiment. And now I have a better question than the one I started with. I don't want to build a bigger graph yet. I want to try to make this result disappear. Better retrieval. More evidence. Interactive retrieval. Different models. Different relational representations. And especially an experiment that removes the “trusted metadata” explanation. If better retrieval makes the relational advantage disappear, that's useful. If it doesn't, that's useful too. Either result tells me where to look next. The interesting question is no longer: **Does GraphRAG work?** That's too broad to be useful. The question I'm interested in is much smaller: **When an LLM already has the relevant documents, does explicitly representing the relationships between those documents give it information it cannot reliably reconstruct from the documents themselves?** I don't have the final answer yet. That's exactly why I'm running the next experiment.