I built a RAG system over six episodes of a company podcast. Small corpus, clean transcripts, a good embedding model, sensible chunking. Then I asked it a question and it lied to me with complete confidence.
Q: Who leads the team responsible for the service that caused the biggest outage?Marcus Chen leads the Platform team, which is responsible for the servicethat caused the biggest outage (ep 3).Retrieved from: [0.446] ep3 #0 The One Where Everything Broke (Marcus, Ravi, Tomas) [0.373] ep3 #2 The One Where Everything Broke (Marcus, Ravi, Tomas) [0.367] ep6 #1 A Year of Lessons (Ben, Priya, Ravi) [0.356] ep5 #1 Data That Actually Gets Used (Lena, Ravi) [0.345] ep2 #2 Inventory at Scale (Dana, Ravi)
The answer is Priya Raman. The system said Marcus Chen.
Look at what it had to work with. Every retrieved chunk is real. The citation to episode 3 is accurate — episode 3 genuinely is about that outage. Marcus is a senior engineer on the Platform team who spends most of that episode explaining the failure in technical detail. Given only those five chunks, “Marcus Chen leads Platform” is a ** reasonable inference**.
This is not a hallucination. It is a correct inference from incomplete evidence, delivered with an honest citation. Which makes it far more dangerous than a hallucination, because nothing about it looks wrong.
Fixing this is not a matter of a better embedding model. It is a matter of choosing a different retrieval architecture — and which one you need depends entirely on why your retrieval is failing. This post is about telling those failures apart.
The chunk naming Priya as Director of Platform Engineering is ep1-000. It was never retrieved.
That single fact settles the diagnosis. This is a retrieval failure, not a reasoning failure. The model never saw the evidence. Swapping GPT-4o-mini for something ten times larger changes nothing — you would be upgrading the reader, not the librarian
Here is the part that made it click for me. Same corpus, same index, same top-k, one different question:
Q: Who is the Director of Platform Engineering?Priya Raman is the Director of Platform Engineering (ep 1).
Instant, correct, first try. The information was sitting in the index the whole time. The first question simply could not reach it.
I ran the numbers on why. Between the failing question and the chunk that holds its answer, the count of shared content words is zero. Not few — zero. “Who leads the team responsible for the service that caused the biggest outage” and “I’m Priya Raman, I’m the Director of Platform Engineering here” have no vocabulary in common at all. The embedding model had to bridge that gap unaided, and episode 1’s team introduction is not semantically about outages.
Vector search doesn’t fail because the answer is absent. It fails because the question is phrased in words the answer doesn’t use.
Once you look for it, vector RAG fails in four recognizable ways:
Vocabulary gap. The question and the answer share no words, and no amount of semantic similarity closes the distance.
Incomplete recall. Top-k is ranked, not exhaustive. It has no concept of “that’s all of them,” so counting questions and “list every…” questions are guesses dressed as answers.
Multi-hop. Every individual fact is in the corpus. The chain connecting them is not in any single passage.
Staleness. Nothing in an embedding says “this fact is retired.” A superseded policy sits in your index scoring beautifully forever.
Before going further: I am not here to talk you out of vector search. On the right question it is excellent, and nothing in this post beats it.
Q: Why did the connection pool exhaust during the outage?The load tests used synthetic traffic that was evenly distributed, becausethat's what's easy to generate. Real warehouse traffic is spiky and clustersaround shift changes — at 06:30 every warehouse in Europe starts its morningshift at roughly the same time. The pool exhausted, requests queued waitingfor a connection, the queue grew, and timeouts cascaded (ep 3).
Perfect answer. One speaker, one turn, one complete causal explanation. That is what vector search is built for.
The principle underneath: vector stores hold language. Opinion, reasoning, nuance, and every “why” answer live in continuous prose. No edge in a graph and no node in a tree can hold five minutes of an engineer explaining why a cutover scared him.
There is also an honest cost argument. Vector RAG is a pip install and an afternoon. Everything else in this post costs days to build and carries ongoing maintenance. If your questions are mostly "what did someone say about X," stop reading and ship it. Adding architecture you don't need is its own kind of failure.
Now change the corpus. Instead of podcast transcripts, imagine a 200-page annual report.
Now change the corpus. Instead of podcast transcripts, imagine a 200-page annual report.
Ask it about revenue. The word “revenue” appears in the MD&A discussion, in three different footnotes, in the risk factors, in the auditor’s letter, and in the segment tables. Similarity search cannot tell you which one your question means, because all of them genuinely are about revenue. You get five chunks that are all topically perfect and contextually wrong.
Chunking is problematic because splitting a number from its footnote destroys the crucial context.
This is a different problem from the Priya problem. Nothing is scattered across documents. Everything you need is in one file. The failure is navigation, not similarity.
is built for exactly this. It is a vector-less, reasoning-based retrieval engine from Vectify AI. Instead of chunking and embedding, it builds a table-of-contents tree from the document, then has an LLM navigate that tree by reasoning — reading section headings, forming a hypothesis about where the answer lives, and drilling down. No embeddings, no chunks, no vector database. It reports 98.7% accuracy on FinanceBench.
The reason it works is that it copies what a human analyst does. You do not scan a filing for semantically similar paragraphs. You go to the section you already know contains the answer. Retrieval becomes structured exploration instead of nearest-neighbour lookup, and every retrieval leaves an auditable path — which matters enormously in regulated domains where “why did you surface this?” is a question someone will actually ask.
Now the honest part. I could not run PageIndex on my corpus, and that failure is the most useful thing I can tell you about it.
Podcast transcripts have no table of contents. No sections, no headings, no hierarchy. There is nothing to build a tree from. PageIndex needs document structure the way vector search needs semantic overlap — it is not an optional input, it is the entire mechanism.
That single fact tells you more about when to reach for it than any benchmark:
Use it when you have long documents with real structure — filings, contracts, clinical protocols, technical manuals — precision matters more than latency, and you need an explainable retrieval path.
Don’t when your text is unstructured (transcripts, chat logs, tickets, email), your facts are spread across many separate documents, or you are on a latency-sensitive path. Tree search costs LLM calls, and they are not free.
Back to Priya, because her problem was never about document structure
Episode 3 says “the platform lead signed off on the rollback at 11:20.” It never names her. Episode 1, recorded six months earlier, says “I’m Priya Raman, I’m the Director of Platform Engineering.” The answer requires connecting two documents that share no vocabulary and never reference each other.
No chunking strategy fixes this. No document tree fixes this — there is no single document to navigate. The information is genuinely distributed.
What fixes it is storing the relationships and walking them:
INC-204 ──CAUSED_BY──▶ scheduler-service │ OWNS ▼ Platform Engineering │ LEADS ▼ Priya Raman
Four hops, one Cypher query, and it returns Priya regardless of how the question was phrased.
Here is the insight that took me embarrassingly long to articulate: an edge has no vocabulary. Once Priya —LEADS→ Platform exists as a stored relationship, it does not matter whether someone asks about "the director," "the platform lead," "who's in charge," or "the team responsible for the biggest outage." The traversal is identical every time. That is the entire fix.
Graphs also do two things nothing else on this list can do.
They answer questions about absence. “Which teams had no customer-impacting incidents?” is unanswerable by retrieval, because you cannot retrieve a non-mention — there is no embedding for nothing. A closed-world graph subtracts one complete set from another and returns an exact answer.
They know what is currently true. This is my favorites example, and the one I’d show a skeptic first:
Q: What is our current deploy policy?VECTOR: There is a deploy freeze policy — no deploys on Fridays (ep 3).GRAPH: Progressive rollout. Every change to a tier-one service goes to 5% of warehouses for 24 hours first. This replaced the Friday deploy freeze, which was retired in December (ep 6).
The vector answer is fluent, correctly cited, drawn from a real chunk — and describes a rule that was killed six months earlier. In the graph, that policy carries status: 'retired', so it is structurally excluded from the result. Not ranked lower. Excluded.
Use Graph RAG when answering needs multi-hop connections across documents, when you need counting or completeness, when facts change status over time, or when entities are referred to inconsistently.
Don’t when your questions are about reasoning and nuance, when your corpus has no stable recurring entities, or when you cannot commit to maintaining the graph — which brings me to the part most posts skip.
Ask about your failure, not your data:
1. Is the answer contained in a single passage of prose? → Vanilla RAG. Ship it. Do not over-engineer.
2. Is the answer inside one long document, but retrieval keeps landing in the wrong section? → Page Index. Your problem is navigation.
3. Does answering require connecting facts across documents, counting things, or knowing what is currently true? → Graph RAG. Your problem is structure.
4. Do you need nuance and structure? →
Everyone sells the upside. Three costs I paid, that you will pay too.
Alias maintenance never ends. I asked my finished system “who signed off on the rollback?” and the graph returned nothing at all. Not a wrong answer — no answer. The reason: “rollback” was not in any entity’s alias list, so entity resolution found nothing, so there was no node to start traversing from. The fix took sixty seconds. Finding it required a real question I had not anticipated. In production, every query where entity resolution comes back empty is an item on a backlog you will be working forever.
Closed-world answers lie with total confidence. “Which teams had no incidents?” only works if the graph is complete. If extraction missed one CAUSED edge, the query returns a clean, authoritative, wrong answer with no hint that anything is off. Graph completeness is answer correctness, and extraction is done by an LLM, which means it is roughly 85% reliable before you add validation.
Empty is not absence — and you have to engineer that in. A wrong Cypher query returns zero rows. Zero rows looks exactly like “no such fact exists.” Left alone, your system will confidently report that an outage never happened when the truth is that you fumbled a relationship name. I ended up making every empty result carry a diagnosis — does the entity exist? does it have any edges at all? — and instructing the answering prompt to say “I could not find that” rather than “that did not happen.”
One more thing worth doing regardless of architecture: measure routing, retrieval, and generation separately. They need opposite fixes, and a single quality score hides which one broke. Doing this is how I found a router bug my system had been silently working around — a retry loop was quietly rescuing a misrouted question on every single run, and the end-to-end output looked fine the whole time.
The question is not which retrieval architecture is best. It is which failure you actually have.
You cannot know that in advance. Build the naive version first, ask it the questions your users will really ask, and let it fail. The shape of the failure tells you what to build next — and quite often it tells you to build nothing, because vector search was fine all along.
The corpus, the graph schema, the LangGraph agent, and the evaluation harness are all in the repo: **GitHub **Link
If you have hit one of these walls, I would like to know which one. My guess is the staleness problem is more common than people realize, precisely because it never looks like a bug.
Vanilla RAG vs PageIndex vs GraphRAG: Which One Does Your Problem Actually Need? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.