Vector Database vs Knowledge Graph: Choosing Your LLM Store

An engineer detailed the trade-offs between vector databases and knowledge graphs for LLM applications, arguing that the choice should be based on query type rather than hype. The article scores both storage paradigms on criteria like retrieval, reasoning, and cost, noting that vector databases excel at semantic similarity while knowledge graphs handle relational queries and multi-hop reasoning. The engineer cited a client case in Mumbai where a vector database failed to answer a multi-condition query that a knowledge graph could handle.

A field guide to the two storage paradigms behind every serious LLM application — scored on retrieval, reasoning, cost, and the questions each one can and cannot answer. A marketplace client in Mumbai came to me with a genuinely good question. Their LLM support agent kept giving almost right answers about their catalog — thousands of products, suppliers, and delivery zones — because the knowledge lived in a vector database that could find "similar" things but could not answer questions like "which suppliers ship to Zone 4 and have a return rate under 3%?" That question requires combining two conditions and traversing a relationship. A vector search cannot do it. A knowledge graph can — and only if it has been built with those relationships in the first place. That client conversation is the entire argument of this article, so let me be direct about the framing: vector databases and knowledge graphs are not interchangeable, they answer different questions, and most teams pick one because it is fashionable rather than because it fits their queries. I have now shipped both in production for clients — Qdrant and pgvector on the vector side, Neo4j on the graph side — and I am going to score them on the criteria that actually decide budgets, not the ones that decide conference talks. Vector databases Qdrant, pgvector, Pinecone, Milvus, Weaviate, Chroma store embeddings — high-dimensional numeric vectors representing meaning. Querying means: embed the question, find the top-k vectors that are nearest in space, return their source text. They are built for semantic similarity . The unit of knowledge is a chunk of text, and the operation is "what is most like this?" Knowledge graphs Neo4j, RDF stores store entities and relationships as first-class citizens — nodes like Supplier , Zone , Product — and edges like SUPPLIES TO , HAS RETURN RATE . Querying means traversing those edges, often with Cypher or SPARQL, and it is exact and deterministic. The unit of knowledge is a fact with a structure, and the operation is "how do these things connect, and which ones satisfy these exact conditions?" The single most useful sentence I tell clients: a vector database answers "what is similar?", a knowledge graph answers "what is related, and how?" If your questions are all similarity — "find relevant policy documents" — you want vectors. If your questions are relational — "which suppliers ship to Zone 4 with return rate under 3%" — you want a graph. Most real applications have some of both. Here is the table I actually use, scored 1–5 from production experience: | Criterion | Vector DB | Knowledge Graph | |---|---|---| | Semantic / fuzzy matching | 5 — the entire point | 1 — exact matching only | | Multi-condition / relational queries | 1 — cannot combine predicates across chunks | 5 — native | | Multi-hop reasoning A→B→C | 1 — not expressible | 5 — first-class | | Freshness / adding knowledge | 4 — embed a new chunk | 2 — new node + edges + schema | | Explainability of answers | 3 — can show source chunk | 5 — can show the exact path | | Setup & ops complexity | 4 — simple to run | 2 — schema design, migrations, Cypher | | Query latency | 4 — tens of ms at top-k | 4 — fast but degrades on deep traversals | | Scale to millions of items | 5 — built for it | 3 — expensive graphs at that size | | Cost to operate | 4 — cheap to start | 2 — memory-heavy, indexes cost | | Ecosystem / tooling maturity | 4 | 4 | | Handling unstructured text | 5 — native | 2 — requires entity extraction first | The asymmetry is stark and it is supposed to be. They are solving different problems. The honest reading of the table: choose on query type, not on hype. Similarity-heavy workloads are vector territory; relationship-heavy workloads are graph territory. The trap is assuming the store you already have can handle the other's job. Where they win. For the workhorse task of LLM apps — retrieval-augmented generation over documents — vectors are unbeatable. Embedding chunks of a policy manual and retrieving the top-5 most relevant at query time is the cheapest way to get an LLM to answer from your current documents, and I have measured it at tens of milliseconds at meaningful scale. Setup is genuinely easy: pgvector runs inside PostgreSQL you already have, and a dedicated store like Qdrant is a weekend to stand up. Cost at scale is friendly because you store one vector per chunk, not a web of edges. Where they disappoint. Vectors are useless at the exact moment you need precision and structure. They cannot do WHERE clauses. They cannot combine "ships to Zone 4" with "return rate under 3%" because each chunk is an island — there is no notion of a supplier having a return rate, or a zone being reachable . Worse, semantic similarity is fuzzy by design: top-k retrieval will happily return a chunk that looks relevant and is factually wrong for the exact question, and the LLM will answer from it. If your workload is "questions with precise, combinable conditions," a vector database will produce fluent-sounding wrong answers, which is the most expensive kind. One concrete case: a logistics client stored their entire rate card as vector chunks. "What does shipping cost from Mumbai to Delhi for a 10kg parcel?" retrieved the closest chunk, which happened to describe a different weight band. The answer was confidently wrong. That was not a bug in Qdrant — it was a category error. The question was relational weight × origin × destination , and the store was built for similarity. Where they win. If your data is relational and your queries are precise, a graph is the only honest answer. That marketplace client's question — "suppliers shipping to Zone 4 with return rate under 3%" — is one Cypher query, exact and reproducible, with the traversed path visible for auditing. Explainability is a genuine superpower: you can show a regulator or a stakeholder the exact nodes and edges that produced an answer, which vector stores structurally cannot. For domains with strong relationships — supply chains, fraud networks, entitlements, org structures — graphs are not a nice-to-have, they are the correct data model. Where they disappoint. You pay for that structure in labor. A graph is only as good as its schema and its construction: you must design node types, edge types, and constraints, and you must extract entities and relationships from your source data before anything works. That entity extraction is real engineering — NLP pipelines, deduplication, and ongoing maintenance when the world changes. Every new relationship you did not model is a query you cannot run. Cost at scale is also honest: graph traversal over millions of nodes with deep hops burns memory and index space, and getting performance right is its own specialization. And for plain semantic search — "find me the documents about refunds" — a graph is awkward, because you have no vectors and no fuzzy matching. Here is the bottom line, and it is not a compromise for its own sake: most serious LLM applications end up needing both, because most real questions mix similarity and relationship. "Find documents about refunds" is a vector question. "Which of those refund policies apply to this specific order type with this specific merchant tier" is a graph question. The practical architecture I ship now is a small one: keep the document corpus in a vector store for retrieval, keep structured entities and relationships in a graph for exact reasoning, and have the agent orchestrate across both. GraphRAG — Microsoft's pattern of building a knowledge graph from a document corpus and using graph traversal plus retrieval together — is the current expression of this hybrid, and it works because it gives the LLM both the relevant text and the exact structure around it. It is more work to build than a plain vector pipeline, but for domains where relationships carry the meaning, it is the difference between an agent that guesses and an agent that knows. I built this hybrid for a compliance client: documents embedded in a vector store for search, regulation entities and their cross-references in Neo4j for "does regulation X apply to workflow Y" questions. Their hit rate on relational queries went from "usually wrong" to "auditable and correct," because the store finally matched the question. The criteria table is the framework, but the tool choice inside each camp matters enough to be explicit about, because I have hit the sharp edges of all of them. In the vector camp. pgvector is my default for teams that already run PostgreSQL — you add a column type, not an infrastructure project, and you keep transactions and backups in one place. It is not the fastest vector index at tens of millions of vectors, but for 90% of apps it is plenty and it is the least operational surface. Qdrant is the right move when you outgrow pgvector : dedicated filtering payload filters that vetores can apply during the ANN search , snapshots, and a cleaner horizontal scale story. Milvus is the heavy artillery for genuinely massive corpora, and it shows in the ops burden — I would not start there. Chroma is fine for prototypes; I have never kept it in production, because its operational story is thin. The honest throughline: start with pgvector , graduate to Qdrant only when measured, not when fashionable. In the graph camp. Neo4j is the pragmatic default — mature Cypher, tooling, and a community you can hire for. The cost is memory: Neo4j wants real RAM, and the licensing model changes the moment you scale past the community edition's limits, so read the fine print before you commit an enterprise. RDF/SPARQL stores are for standards-obsessed domains government, linked-data ; they are powerful and genuinely unpleasant to work with. The forgotten option is a hybrid inside one database — pgvector in Postgres, plus Postgres itself storing normalized tables you can JOIN for relational queries. It is not a real graph no multi-hop traversal engine , but it covers a surprising share of "relational" needs before you are ready for a dedicated graph. The measured truth: for the marketplace client, the vector-only agent answered "which suppliers ship to Zone 4" wrong four times out of five; after adding a graph for the structured part, the same question answered correctly with a visible path. The improvement was not a better vector model. It was a better question-to-store match, which is the entire lesson of this article. When a client asks me vector DB or knowledge graph, I run them through four questions. Write them down; they decide the architecture: pgvector if you want zero new infrastructure. You are done until question 2 starts hurting.The failure I see most often is not picking the wrong store — it is picking one store because the other was never considered, and then discovering six months later, when a client asks a question with a WHERE clause, that the entire knowledge layer is built on islands of similar text. Ask the four questions before you pick, and you will pick right. Gulshan Yad