Traditional RAG retrieval fails on the questions that matter. Knowledge graphs answer them at 1000x the cost. We shipped the middle path.
👋 Hey, Jai here! I’m a software engineer and tech investor turned solo founder building a bootstrapped software company. I write about using AI to build faster, sell smarter, and run a whole company solo. New here? Join me:
Or if you don’t like my work, send me hate mail.
Almost every business I’ve spoken to at QX has asked for some version of an AI “company brain” connected to their internal documents and systems. They echo the same complaint that their data is scattered across silos, and believe that the key to their competitive edge lies in unlocking this pent-up institutional knowledge.
In a world where AI is commoditising access to public knowledge, internal knowledge accumulated through domain experience and relationships become the primary moat. Two years ago, the AI brain discussion warranted an exploratory pilot, but today it’s table stakes.
The common way to build this is called RAG, or retrieval-augmented generation. This process involves breaking your company’s documents into passages, converting them into numerical representations that encode their meaning, and letting an AI answer questions by retrieving the passages that best match a query. It’s simple, cheap and demos well.
RAG was originally proposed by Meta AI researchers in a May 2020 paper. In AI-years that’s almost pre-historic. And whilst it performed well on a limited set of questions, today it falls short for use-cases where real value is added. Microsoft then entered the frame in 2024 and published a technique called GraphRAG. Their idea was to create a knowledge graph by using LLMs to extract the entities and relationships from documents, not just the text content. This would enable us to traverse a structured graph during retrieval (e.g. “find me documents related to this person, company, event etc”) instead of a simple text search. It gained a lot of hype.
I’ll get into the detail of GraphRAG shortly, but my view is that in practice it’s complete overkill. At QX, with the benefit of experience and testing, we have chosen a more pragmatic approach: take the best parts of GraphRAG and knowledge graphs, fit them into a much simpler architecture, and scrap the rest. Our approach has been shaped by what actually kills these AI brain projects in practice:
Information is unstructured and spread across disconnected systems.
Retrieval quality has to sustain with scale, i.e. beyond a pilot.
Initial implementation needs to be fast and cheap, so that companies can see results and drive uptake before committing outright. Behavioural change is
alwaysthe primary goal, everything else follows.
Crucially, we wanted a system that our mid-market customers could switch on themselves, without consultants or setup workshops.
Three kinds of questions #
While analysing the queries customers sent to their internal knowledge base, we identified three distinct categories. Needle****questions.“What was the take rate in the 2017 valuation memo?” These typically ask for one fact in one document. Classic RAG is excellent here because searching by meaning finds that one passage reliably.Portfolio****questions.“Tell me everything we know about Acme Inc.” This is a broad, often multi-step question that needs every relevant document, not the top ten passages. RAG gets you a decent sample but silently drops the rest which is dangerous for diligence or research work (creates blind spots).Counting questions.“Which fintech companies have we evaluated?” or “Which contracts mention this supplier?” These need an exact list or tally across the entire corpus. Top-k search results cannot answer them. The system will confidently name the companies it happened to retrieve and skip the rest, and you’ll never know.
Most “AI brain” demos show needle questions, but real business value is often trapped behind the other two.
Enter GraphRAG #
GraphRAG took direct aim at solving for the limitations of classic RAG. Instead of just chopping documents into searchable passages, Microsoft proposed that an AI should parse the whole corpus and build a knowledge graph out of it. Every company, person, and product becomes a node. Relationships between them become edges. Clusters of related entities get pre-written AI summaries, so the system can answer big-picture questions from the summaries instead of hunting through passages.
It’s a clever idea and it works well for those latter question categories. The research is consistent: graph-based approaches shine on questions that require connecting information across many documents, what the original paper called “global sensemaking”.
GraphRAG became shorthand for “the serious way to do enterprise RAG”. But it came at a real cost.
What the research says #
There were three major findings when evaluating GraphRAG, and most of them came directly from Microsoft.
The index is very expensive. Building the graph means running AI extraction and summarisation over everything before anyone asks a single question. Microsoft’s own follow-up work, LazyGraphRAG, states the cost: a plain vector index costs about 0.1% of a full GraphRAG index. Three orders of magnitude, paid up front, on content nobody may ever query. The same work showed a lazy approach matching full GraphRAG’s answer quality on global questions at a small fraction of the query cost. In effect, Microsoft showed that the expensive part of GraphRAG is mostly unnecessary. Their productised solution accelerator for GraphRAG was even archived in 2025.
The wins are narrow. A systematic evaluation published in 2025 (RAG vs. GraphRAG, arXiv:2502.11371) found what practitioners kept rediscovering: vanilla RAG wins on direct factual questions, graphs win on multi-hop and summary questions, and the best results come from routing between approaches rather than betting on either.
Entity resolution is the hardest part. For a graph to be useful, “Acme”, “ACME Holdings Inc.” and “Acme Corp” must become one node. Get that wrong in one direction and your graph fragments into disconnected shards. Get it wrong in the other direction and two different companies inconspicuously merge into a single fictional one, and every answer built on top inherits the error. This problem is old and largely unsolved in the popular GraphRAG tooling. Meanwhile a graph database brings its own operational tax: another system to run, back up, secure, and keep consistent with your documents, multiplied by every tenant if you’re a SaaS product.
The philosophy behind GraphRAG is right: documents alone aren’t enough, and tracking the entities that your corpus talks about unlocks question types that ordinary search can’t address. The architecture most people associate with it, an eagerly built knowledge graph in a graph database, is overkill for the majority of real workloads.
What we built instead: graph-like RAG #
We kept the parts with evidence behind them and dropped the rest. The result looks like a graph but runs on a regular database: it has entities, relationships, and rich metadata that evolve over time—but no Neo4j, no up-front graph build, and no ontology to custom-design.
The breakdown is as follows:
Everything stays in ordinary infrastructure. Passages live in a search index. Entities live as plain database records: one record per company, person, product, project, event, or location, per client workspace. An entity record knows its name, its alternate spellings, which documents mention it, and which other entities it tends to appear alongside. Those “appears alongside” links are what effectively form the graph. They’re just a list stored on each record, computed by counting co-occurrences. There’s no graph query language and no new moving parts.
The ontology is fixed, small, and universal. Six types (company, person, event, etc.) plus a handful of label fields (industry, category, topic) that adapt to whatever the documents contain. The fixed ontology is what makes the system self-serve. Custom ontologies are otherwise why traditional knowledge-graph projects begin with months of workshops. A pharma client and a private equity client get the same types, and the content of the graph adapts to their world automatically, because it’s extracted from their documents and evolves naturally as the corpus grows.
Entity resolution follows a clear waterfall. When a document mentions a name, we check a “phone book” containing every spelling we’ve seen before for that entity. If there’s an exact hit, the entity is matched for free. If we miss, a similarity search proposes look-alike candidates for fractions of a cent. Only when there’s a genuinely ambiguous candidate does a small AI model get asked “are these the same real-world thing?”, and it’s instructed to say no when unsure. A wrong merge poisons everything downstream; a missed merge just leaves two entries that a recurring cleanup job can unite later. Every merge is reversible by design.
Entity summaries are built “lazily”. Nothing gets an AI-written profile until someone actually asks about it. This is the lesson from Microsoft’s LazyGraphRAG work. We want costs that scale with usage instead of costs that scale with the size of the corpus.
The AI decides what retrieval strategy (graph vs text) to use for a given query. The agent answering your question has plain search for needle questions, a “resolve” tool that pulls everything about one entity, “expand” that walks to related entities (similar to “hopping” a graph database), and exact counting over the whole corpus for list questions. You can watch it chain these together. See the example below: we ask a broad portfolio-style question. It first uses facet to count entities (in this case companies) that get mentioned frequently in the context of food delivery, and then resolves each entity to dig deeper.
Does it work? #
I’m obsessed with measurement, so before building any of this we developed an evaluation harness: a corpus of thousands of real documents and a graded set of questions across all three question types, scored automatically. Every architectural decision above had to move a number or it didn’t ship. In the end, the entity layer produced a clear uplift on exactly the question classes where plain RAG fails (portfolio and counting questions) while leaving needle questions at ceiling. The eval system deserves its own write-up so maybe I’ll get to that in a future post.
The trade-offs #
It doesn’t do deep multi-hop reasoning. If your use case is “find the hidden path between these two entities four steps apart”, you want a real graph that can handle complex traversal.
Our links are co-occurrence, not typed relationships: the system knows that the company Acme and the person Jane Smith appear together constantly, but not that she’s the CEO. Our thesis is that, once the AI is supplied with the surrounding context for these two entities, it can gracefully deduce that Jane is the CEO without having to encode this in the graph itself.
Our conservative merging also means an occasional entity exists twice for a day until the cleanup job unifies it.
If your corpus is small and static, and your questions require you to connect tenuous dots across the graph, full GraphRAG is a good choice and the costs are probably worth it. Our bet is different: for working companies with living document sets, the graph-like version captures most of the value at a fraction of the cost and none of the operational weight.
The self-serve factor #
Everything above runs self-serve. Connect Google Drive, SharePoint, Notion, Granola etc, and decide how frequently you want them to sync. The system extracts, resolves, links, and trues itself up on a schedule, and the costs of that upkeep scale with what changed, not with how much you’ve stored. Nobody designs a schema or labels anything. That was the constraint we started with, and I believe it was the best design decision we made.
This is how we approached it at QX Labs. If you’re wrestling with the same problems, or you think we’re wrong somewhere, I’d love to exchange notes.
Sources and further reading:
Lewis et al.,
(arXiv:2005.11401), the first paper to propose RAG[Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)Edge et al.,
(arXiv:2404.16130), the original Microsoft GraphRAG paper[From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://arxiv.org/abs/2404.16130)Microsoft Research,
(Nov 2024), source of the 0.1% indexing cost and 700x query cost comparisonsLazyGraphRAG: Setting a new standard for quality and costHan et al.,
(arXiv:2502.11371)RAG vs. GraphRAG: A Systematic Evaluation and Key InsightsAnthropic, (2024), the highest-evidence cheap retrieval upgrade we adopted, measured at a 49% reduction in retrieval failuresIntroducing Contextual Retrieval