# Why Vector Databases Are Critical for RAG Systems

> Source: <https://dev.to/mryadavgulshan/why-vector-databases-are-critical-for-rag-systems-a9g>
> Published: 2026-08-27 02:30:00+00:00

*A practical look at the retrieval layer of RAG — what a vector database actually does, why keyword search fails, and what it costs to get it wrong.*

A few weeks ago, a health-tech client sent me a screenshot that made me stop typing. His RAG system had answered a doctor's question about a medication interaction with a dosing suggestion that was wrong — not subtly wrong, dangerously wrong. The team's first instinct was to blame the model. I asked one question: "How is your system retrieving the context?" Nobody knew. We opened the code and found it: keyword search over their knowledge base, feeding the top hits into the prompt. There was no vector database anywhere in the pipeline.

That answer explained everything. The model had done its job. It read three irrelevant chunks and produced a confident, grounded-sounding, completely wrong answer. The retrieval was broken — there was no way for it to find meaning, only matching strings.

That incident is why I am writing this. Vector databases are not a trendy accessory you bolt onto an LLM project. They are the difference between a RAG system that quotes the right source and one that confidently quotes the wrong one. This article covers what a vector database actually does under the hood, the taxonomy of options, a production architecture, real code, and the honest failure modes — including when you should not use one at all.## The Problem Keyword Search Can't Solve

To understand why vector databases exist, you have to understand the exact failure that killed my client's system. Keyword search — BM25, SQL `LIKE`

, Elasticsearch — matches exact strings. It is brilliant at that, and terrible at meaning.

Consider two sentences:

Keyword search sees almost no shared tokens between them — "lower" and "reduced" match, nothing else does — so it scores them as near-total strangers and drops the chunk that actually contains the answer. The retrieval layer hands the LLM the wrong documents, and the LLM makes the best of garbage.

The deeper point: in RAG the retrieval step is the source of truth the model is bound to. When the retrieved context is correct, the model will not invent facts it cannot support — but when the context is wrong, it will not know it is wrong either. Retrieval quality sets a hard ceiling on the whole system, and no amount of prompt engineering, fine-tuning, or fancier models raises it. They only make the errors sound more confident.

That is the entire argument for vector databases in one paragraph: search by meaning instead of by exact string.## How Embeddings Actually Work

A vector database stores vectors, and vectors come from embedding models. Here is the mechanism in one paragraph: an embedding model converts a piece of text into a list of numbers — usually 768 to 3,072 floats — arranged so that semantically similar texts land close together in that high-dimensional space. "The dog chased the ball" and "the canine ran after the toy" produce vectors that are nearly parallel, even though they share almost no words. Distance in that space is a proxy for difference in meaning; the famous toy example — `king − man + woman ≈ queen`

— is the same idea in arithmetic form.

The critical practical detail is that the embedding model you use is a permanent architectural decision. You index documents with it, then you query with it. Change the model later and every vector in the store is in the wrong language for the new model — you must re-embed the entire corpus. Decide early, measure the retrieval quality on your own data, and treat the choice as locked-in until you have a hard reason to pay for a re-embed.

Here is the part that confuses people: the vector database is not doing anything clever with meaning. The embedding model produced the meaning; the database just makes searching it fast and correct. Its job is to solve one specific problem — finding the k nearest neighbors of a query vector among millions of vectors, in milliseconds, at scale.

Finding the exact nearest neighbor in a million-vector space is too slow for real-time retrieval. So vector databases use Approximate Nearest Neighbor (ANN) algorithms that trade a small amount of accuracy for orders of magnitude in speed. The two you will actually meet:

The practical guidance I give clients: start with HNSW — easier to tune, faster to query, and your corpus needs to be very large before memory cost becomes the thing that matters.

The measure of "near" is configurable, and getting it wrong silently degrades retrieval. The three you will see:

Rule of thumb: if your embedding model normalizes its output vectors to unit length, dot product and cosine give identical rankings, and dot product is cheaper. If unsure, use cosine.

Here is the requirement most tutorials skip and every production system needs: you almost never want to search the whole corpus — you want "all support tickets from the last 90 days" or "policies that apply to the EU region." A vector database that cannot filter by metadata during the ANN search forces you into a classic failure — pulling every matching vector into memory, then doing an exact search over them, which blows your latency budget on a large corpus.

The right approach is index-level filtering: store region, date, document type, tenant ID, and permissions as payload fields, and let the database apply them inside the search. This is also where multi-tenancy lives — one shared store serving ten companies with no tenant filter is an information-leak bug waiting to happen. Filter by tenant, always.

Every week someone asks me which vector database to use. Here is the landscape as I actually reason about it.

| Option | What it is | Best for | Watch out for |
|---|---|---|---|
Qdrant |
Dedicated vector DB, Rust, HNSW-first, metadata filtering first-class | Production RAG, multi-tenant, needs fast filtered search | One more service to operate |
pgvector |
Postgres extension | Teams already on Postgres, small-to-medium corpora, simplest ops | Filtered search gets slow past a few million vectors |
FAISS |
Library, not a server (from Meta) | Offline indexing, research, embedding pipelines at scale | You build the persistence, filtering, and serving yourself |
Managed (Pinecone, Weaviate Cloud, etc.) |
Hosted vector DBs | Teams that want zero ops | Cost grows with volume and latency guarantees vary |

The honest advice: if you are already running Postgres and your corpus is under a few million chunks, `pgvector`

is the right first move — one fewer moving part, and your transactions and vectors live in one place. The moment filtered search on a large index starts hurting, or you need predictable multi-tenant latency, move to a dedicated engine like Qdrant. FAISS is for building pipelines, not shipping apps; managed services are for teams whose scarce resource is time, not money.

Let me make this concrete. Here is the smallest real architecture I would ship: embed with a good model, store in Qdrant, search with metadata filtering, feed the top chunks to an LLM. I will use the Qdrant client with an OpenAI-compatible embedding endpoint.

First, create the collection with the right distance function and a payload schema that carries metadata:

``` python
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, PointStruct

client = QdrantClient(url="http://localhost:6333")
COLLECTION = "knowledge_base"

client.recreate_collection(
    collection_name=COLLECTION,
    vectors_config=VectorParams(size=1024, distance=Distance.COSINE),
)
```

Then index a batch of documents, storing the text alongside its vector so you can return the actual source, not just an ID:

``` php
import requests

def embed(text: str) -> list[float]:
    r = requests.post(
        "http://localhost:8000/v1/embeddings",
        json={"model": "your-embedding-model", "input": text},
    )
    return r.json()["data"][0]["embedding"]

docs = [
    {"id": 1, "tenant": "acme", "region": "EU", "text": "Kidney-impaired patients should receive a reduced dose."},
    {"id": 2, "tenant": "acme", "region": "EU", "text": "Standard adult dosing for this compound is 40 mg once daily."},
]

points = [
    PointStruct(
        id=d["id"],
        vector=embed(d["text"]),
        payload={"tenant": d["tenant"], "region": d["region"], "text": d["text"]},
    )
    for d in docs
]
client.upsert(collection_name=COLLECTION, points=points)
```

Query with the tenant filter applied at the index level, not as an afterthought:

```
query = "What dose should be used for a patient with reduced kidney function?"

hits = client.search(
    collection_name=COLLECTION,
    query_vector=embed(query),
    query_filter={"must": [{"key": "tenant", "match": {"value": "acme"}}]},
    limit=5,
)

context = "\n\n".join(h.point.payload["text"] for h in hits)
print(context)
```

That `query_filter`

line keeps tenant A's answers from leaking into tenant B's. It belongs inside the search — the reason you want a database that supports index-level filtering rather than a raw library.

The full RAG call then becomes: embed the user question, retrieve the top-k chunks with the tenant filter, join them into a context block, and send that plus the question to the LLM with an instruction to answer only from the provided context.

I have hit every one of these in real deployments, in the order they hurt:

**Chunk size and overlap are treated as defaults.** Chunking determines what retrieval can even find. Chunks too large dilute the meaning per vector; too small lose the surrounding context. There is no universal answer — it depends on your documents — so measure it, don't inherit a magic number from a blog post. Start at 512–768 tokens with 10–15% overlap and benchmark.

**top-k set once, never revisited.** A `limit=3`

that worked on your demo corpus will starve a production corpus. Monitor how often the correct answer appears in the retrieved set and tune k until retrieval quality stops improving.

**No hybrid search, so exact terms get lost.** Embeddings are bad at exact strings: part numbers, error codes, model names. A query for "ERR-1042" will return semantically similar-looking noise instead of the document that literally contains "ERR-1042". The fix is hybrid retrieval — run BM25 keyword search and vector search in parallel and merge results with a reciprocal rank fusion.

**Re-ranking is skipped for cost reasons.** Retrieving 50 candidates and re-ranking the top 20 with a cross-encoder beats just taking the top 5 from vector search. It costs latency and money, and on real user queries it is usually worth it — the single biggest retrieval quality upgrade I can make for a client.

**Embedding drift after a model change.** Upgrade the embedding model and forget the re-embed, and every query is measured against vectors in a different space. Retrieval silently degrades. Version your embeddings and re-embed before you ship.

**The empty-result failure is unhandled.** When retrieval returns nothing relevant, many systems still feed an empty or irrelevant context to the model, which then answers from its training data as if grounded. Enforce a minimum relevance threshold and let the system say "I don't have the information" instead of hallucinating.

The honest part. A vector database is not always the answer, and I have told clients so to their face:

`pgvector`

covers your volume, do not spin up a second database just to feel modern.
## The Practitioner's ChecklistBefore you trust a RAG system, walk this list:

Back to the health-tech client. We replaced the keyword layer with a vector store, added metadata filters so each hospital's data stayed isolated, tuned chunking against their real documents, and re-ranked the top candidates. The wrong-dosage answers stopped because retrieval started returning the right source. We changed the model nowhere and almost nothing about the prompt — the fix was entirely in the retrieval layer.

That is the whole lesson, and I will repeat it every time someone blames the LLM: in RAG, the model is only as good as the context you hand it, and the context is only as good as retrieval. Build the retrieval layer well — it is the difference between sounding grounded and being grounded.

Start small: get your documents into a vector index, add the tenant filter, and benchmark before you buy a second server. The day a wrong chunk reaches a clinician, the vector database was never a convenience — it was the floor.

*Gulshan Yad