Part 7 of the Production RAG Series — Why Traditional Databases Crumble Under Vectors, How Vector DBs Actually Store Your Data, and the IVF Index That Makes Search Possible.
Hey friends! Welcome back to our ongoing series on building production-grade Retrieval-Augmented Generation (RAG).
In Part 6, we had a beautiful breakthrough moment. We learned that instead of matching words letter-by-letter, we can convert entire sentences into GPS coordinates in “meaning space” — called embeddings. Two sentences that mean the same thing land close together on the map, even if they share zero words.
We ended Part 6 standing at the edge of a cliff.
We had our vectors. We had our meaning. But we asked a terrifying question:
“Now what? Where do we actually store these millions of vectors? And how on earth do we search them in milliseconds?”
You might have had a completely reasonable thought:
“Okay, I already have a MySQL database running. Can I just add a column called embedding and store the numbers there?"
Totally fair. Most engineers’ first instinct is exactly this. And honestly? It works — for a weekend project with 500 rows.
But the moment your company’s knowledge base grows to 1 million chunks, that instinct quietly transforms into a production nightmare.
To understand why, we need to go back to something you already know well: traditional databases.
If you’ve spent any time building software, you’re comfortable with databases. You know the classics:
Relational Databases (SQL): PostgreSQL, MySQL, SQLite. Your data lives in neat rows and columns. You run queries like:
SELECT * FROM orders WHERE customer_id = 42 AND status = 'pending';
The database finds your answer in milliseconds because it built an index — a sorted shortcut that lets it skip 99% of the data.
Each of these databases was brilliantly designed for a specific shape of data and a specific type of question:
Now, traditional indexes like B-Trees aren’t designed for this last question. You need a different kind of index — one that understands distances between vectors.
Let’s get concrete. Imagine you just finished Part 6. You have embeddings for 1,000,000 document chunks. Each embedding is a list of 1536 numbers.
Your natural instinct is to create a table:
CREATE TABLE document_chunks ( id SERIAL PRIMARY KEY, content TEXT, embedding FLOAT[1536] -- a list of 1536 numbers);
Now a user asks: “What is the refund policy?”
You convert their question to a vector [0.34, -0.71, 0.82, ...] and you want to find the 5 most similar rows.
Traditional database’s response? Something like a shrug.
Here’s why.
Imagine you’re looking for the word “Robot” in a physical English dictionary.
You don’t start at page 1 and read every single word. You flip to the “R” section instantly. Why? Because the entire dictionary is arranged in one straight line of order: A to Z.
Traditional database indexes (the famous B-Tree index) work on this exact same genius:
Because everything is on one sorted line, the database can skip the parts it doesn’t need. With 1,000,000 rows, it might check only 20 to find your answer. That’s its superpower.
Now look at this vector:
Vector A: [ 0.82, -0.15, 0.44, 0.19, 0.67, ... ] (1536 numbers)Vector B: [ 0.81, -0.12, 0.50, 0.08, 0.71, ... ] (1536 numbers)Vector C: [ 0.79, -0.30, 0.41, 0.22, 0.55, ... ] (1536 numbers)
Ask yourself a simple question: How would you put these vectors in order from “smallest” to “largest”?
There is no single “A to Z” line in 1536-dimensional space.
You could technically sort vectors by their values, but that wouldn’t tell you which vectors are closest to each other.
That’s why a traditional B-Tree index isn’t suitable for efficiently finding the nearest vectors. It has no natural way to skip large portions of the dataset based on geometric distance.
Without an index to skip rows, the database is forced to do what you’d have to do with a dictionary printed in random order: read every single page.
For 1,000,000 vectors with 1536 dimensions each:
For every user query: Step 1: Compare the query against Vector 1 (1536 dimensions). Step 2: Compare the query against Vector 2 (1536 dimensions). Step 3: Compare the query against Vector 3 (1536 dimensions). ... Step 1,000,000: Compare the query against Vector 1M. Then identify the top 5 highest-scoring matches.
That’s 1.536 billion dimension-wise comparisons for a single query!
And remember, this is just one user searching. Multiple simultaneous queries multiply the computational workload.
A Vector Database is a specialised piece of infrastructure with one core job: store high-dimensional vectors and retrieve the most similar ones in milliseconds, even at the scale of hundreds of millions of vectors.
Think of it this way:
A regular database is like a city with a strict address system — every building has a number, every street has a name, sorted and indexed so you can jump straight to the right door. Perfect when you know the exact address.
A vector database indexes is like a city that has already mapped every building based on its location. Legal firms might end up clustered in one district. Hospitals form their own neighborhood. Tech companies live somewhere else entirely.
When you arrive asking a question, the city already has a map that helps it identify which district is most likely to have your answer — without you knowing the exact address at all.
But before we get into how this magic works, we need to understand how vectors are stored in the first place.
When you index a document chunk into a vector database, three things get stored together:
┌───────────────────────────────────────────────────────────────────┐│ STORED RECORD (One per chunk) ││ ││ 1. ID: chunk_1847 ││ ││ 2. VECTOR: [ 0.34, -0.71, 0.82, 0.15, -0.09, 0.63, ... ] ││ (The "GPS coordinate" of this chunk's meaning) ││ ││ 3. PAYLOAD / { ││ METADATA: "source": "employee_handbook.pdf", ││ "page": 42, ││ "section": "Parental Leave Policy", ││ "text": "Employees are entitled to 12 weeks..." ││ } │└───────────────────────────────────────────────────────────────────┘
The vector is the searchable GPS coordinate. The payload contains information associated with that vector — this could include the original text, document name, page number, section, or other metadata.
Here’s the full ingestion flow before anything lands in the vector database:
Each chunk lives as its own record. The original document might become 80 chunks; each chunk gets its own GPS coordinate. The vector database stores all 80 records and builds an index that makes searching through them almost instant.
This is the heart of the matter. Every vector database builds a special index structure on top of your stored vectors. This index doesn’t sort vectors in a line (remember, that’s impossible). Instead, it builds a clever map of the neighborhood so that when a query arrives, the database can jump directly to the relevant area instead of searching everywhere.
There are several types of indexes, each with different trade-offs. Let’s explore them.
Here’s where things get genuinely clever.
Imagine your 1,000,000 document vectors are 1,000,000 people scattered across a huge city. They’re not random — people with similar interests naturally live near each other. The “HR Policy” people cluster together on one side of the city. The “Financial Reports” people form their own neighborhood. The “Tech Docs” people live somewhere else entirely.
Now, when a query arrives, instead of knocking on every door in the city, what if we could first figure out which neighborhood the answer is probably in — and only search there?
That’s exactly what IVF does. And it works in two phases.
IVF runs an algorithm called K-Means clustering on all your vectors. K-Means does one job: group similar vectors into K neighborhoods. Let’s say K=3 for simplicity.
Imagine we have 9 vectors plotted in 2D space (in reality they’d be 1536D, but 2D lets us actually see what’s happening):
Now you might wonder: How does K-Means actually figure out that A, B, and C belong together in Cluster 1, while E, F, and G belong in Cluster 2?
It doesn’t happen by magic. K-Means follows a simple, repeatable loop:
Once it stabilizes, each cluster has its permanent members and its final centroid (the average coordinates of its members):
Cluster 1 (HR): A(1,8), B(2,9), C(2,6) Centroid = ( (1+2+2)/3, (8+9+6)/3 ) = (1.67, 7.67)Cluster 2 (Tech): E(6,7), F(7,8), G(6,6) Centroid = ( (6+7+6)/3, (7+8+6)/3 ) = (6.33, 7.0)Cluster 3 (Finance): D(3,3), H(4,2), I(3,1) Centroid = ( (3+4+3)/3, (3+2+1)/3 ) = (3.33, 2.0)
That’s your index. The K-Means run was expensive, but it happens only once at build time.
Now a query comes in: “What is the parental leave policy?” Query vector: (2.1, 7.5).
Step 1: Compare query to ALL centroids (only 3 comparisons — blazing fast)
Distance from query (2.1, 7.5) to: C1 (1.67, 7.67) = sqrt( (2.1-1.67)² + (7.5-7.67)² ) = sqrt( 0.18 + 0.03 ) = 0.46 ← CLOSEST ✅ C2 (6.33, 7.0) = sqrt( (2.1-6.33)² + (7.5-7.0)² ) = sqrt( 17.9 + 0.25 ) = 4.26 C3 (3.33, 2.0) = sqrt( (2.1-3.33)² + (7.5-2.0)² ) = sqrt( 1.51 + 30.25 ) = 5.63
Cluster 1 wins because its centroid is closest to the query vector. In our illustrative example, this cluster happens to contain the HR-related chunks.
Step 2: Search only Cluster 1 (only 3 vectors: A, B, C)
Distance from query (2.1, 7.5) to: A (1, 8): sqrt( (2.1-1)² + (7.5-8)² ) = sqrt( 1.21 + 0.25 ) = 1.21 B (2, 9): sqrt( (2.1-2)² + (7.5-9)² ) = sqrt( 0.01 + 2.25 ) = 1.50 C (2, 6): sqrt( (2.1-2)² + (7.5-6)² ) = sqrt( 0.01 + 2.25 ) = 1.50
Return: Vector A — closest match. ✅
We did 3 + 3 = 6 comparisons instead of all 9. At real scale with 1,000,000 vectors and 1,000 clusters, we may perform roughly 2,000 vector-distance calculations instead of 1,000,000. That’s a 500x reduction in the number of vectors examined in this simplified example!
You can see the fundamental advantage: instead of searching the entire dataset, IVF narrows the search to a much smaller portion of it.
There’s one catch. What if the real best-matching vector sat right at the boundary between Cluster 1 and Cluster 2? With nprobe=1 (search only 1 cluster), you'd miss it.
nprobe controls how many clusters you search:
nprobe = 1 → Search top 1 cluster → Fastest, slight risk of missing edge casesnprobe = 5 → Search top 5 clusters → Good balance, catches most boundary casesnprobe = 20 → Search top 20 clusters → High accuracy, approaching brute-force speed
Query (2.1, 7.5) │ ┌─────────┼──────────┐ ▼ ▼ ▼ C1: 0.46 C2: 4.26 C3: 5.63 ← distances to centroidsnprobe=1 → Search only C1 (fastest)nprobe=2 → Search C1 and C2 (catches boundary vectors between HR & Tech)nprobe=3 → Search all 3 clusters (same as brute force for this tiny example)
There is no universal sweet spot for nprobe.
The right value depends on your dataset, the number of clusters, and how much recall you’re willing to trade for speed. In practice, engineers benchmark different nprobe values to find the right balance between search quality and query latency.
IVF introduced us to a philosophy that every modern vector database lives by:
Trading some recall for a significant improvement in search speed.
Being 99% accurate and returning results in 5ms is infinitely more useful than being 100% accurate and making users wait 15 seconds. Your users won’t notice the 1% difference. They will absolutely notice the 15-second wait.
This philosophy has a name in computer science: Approximate Nearest Neighbor (ANN) search. It’s the backbone of every production vector database. Every index type we’ll look at — including the gold standard we’ll cover in Part 8 — is a different implementation of this same idea.
IVF is smart, but it has a limitation.
When a query vector falls right on the boundary between two clusters, IVF with a low nprobe can get confused. It picks the closest centroid, searches that cluster, and confidently returns a result — even though the true best match was sitting just across the boundary in the neighboring cluster.
It’s like a city with separate neighborhoods. You’re looking for a restaurant, and the best one is just across the boundary. But because you only searched one neighborhood, you missed it.
In Part 8, we’ll explore a completely different approach — instead of dividing vectors into clusters, what if we connected them through a network of nearby neighbors?
It’s called HNSW (Hierarchical Navigable Small World) — the index that powers Qdrant, Weaviate, Milvus, and pgvector under the hood.
And understanding how it builds and searches this graph, from the moment a vector is inserted right through to a live query, is genuinely one of the most satisfying ideas in modern AI infrastructure.
See you in Part 8! 🚀
⬅️ Previous: Part 5 — Advanced RAG Chunking: Hierarchical, Semantic & Contextual Retrieval
⬅️ Previous: Part 6 — From Keywords to Meaning: How Embeddings Power Semantic Search
📍 You Are Here: Part 7 — The Library That Thinks in Meaning: A Deep Dive into Vector Databases
➡️ Part 8 (Coming Next): The Highway Through High-Dimensional Space: How HNSW Powers Modern Vector Search
The Library That Thinks in Meaning: A Deep Dive into Vector Databases was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.