{"slug": "vector-databases-deep-indexing-token-economics-the-complete-story-phase-3", "title": "Vector Databases, Deep Indexing & Token Economics: The Complete Story (phase 3)", "summary": "A developer explains the architecture of vector databases, covering schema design, indexing methods like IVF and HNSW, product quantization, distance metrics, metadata indexing, and token economics to avoid redundant costs. The post details how to store embeddings efficiently in PostgreSQL with pgvector and discusses trade-offs in vector database selection.", "body_md": "*From \"we have vectors\" to \"this actually scales, and doesn't bankrupt us.\"*\n\n👦 **Nephew:** Uncle, Phase 2 is done. I understand tokenization, embeddings, cosine similarity. But two things are bugging me. First — where do these vectors actually **live**, physically? Second — when we have 5 million of them, how does search stay fast without checking every single one? It feels like magic right now.\n\n👨🦳 **Uncle:** Good — because it's not magic, and if you treat it like magic, you'll make bad decisions later. Which index to use, how much memory you'll need, why your search suddenly got slower after adding a million vectors, why your embedding bill is way higher than it should be — none of that makes sense until you understand what's actually happening underneath. Let's take this in order: first where things live, then how search stays fast, then — the part almost every tutorial skips — how you stop burning money doing any of this.\n\n```\nPhase 2: Embeddings & Semantic Search ✅\n    ↓\nTODAY ← WE ARE HERE\n    │\n    ├─ Part 1: Where Embeddings Actually Get Saved (Schema Design)\n    ├─ Part 2: Why You Can't Just \"Check Everything\" (The Brute-Force Wall)\n    ├─ Part 3: IVF — The Neighborhood Approach\n    ├─ Part 4: HNSW — The Highway Approach\n    ├─ Part 5: Product Quantization — Compressing Vectors\n    ├─ Part 6: Distance Metrics — What \"Similar\" Actually Means\n    ├─ Part 7: Metadata Indexing — The Other Half Everyone Forgets\n    ├─ Part 8: Putting It Together in pgvector\n    ├─ Part 9: Choosing a Vector Database, Honestly\n    └─ Part 10: Token Economics — Stop Paying Twice for the Same Thing\n    ↓\nA System That Survives Real Traffic and a Real Bill\n```\n\n👨🦳 **Uncle:** Before we talk about indexing, let's settle the basics. A vector by itself is useless. What you actually store is a **row** — the vector, plus everything needed to use it later, find it again, and avoid paying for it twice.\n\n**Table: document_chunks**\n\n| Column | Purpose |\n|---|---|\n`id` |\nUnique identifier |\n`chunk_text` |\nThe original text — kept as insurance for re-embedding with a new model later |\n`embedding` |\n`vector(1536)` |\n`embedding_model` |\ne.g. `'text-embedding-3-small'` — different models produce incompatible vector spaces |\n`content_hash` |\nSHA-256 of `chunk_text` — used for dedup, more on this in Part 10 |\n`metadata` |\n`jsonb` — `{ department, doc_type, uploaded_by, source_file }`\n|\n`created_at` |\nTimestamp |\n\n👦 **Nephew:** Why is metadata a separate `jsonb`\n\ncolumn instead of just... more columns?\n\n👨🦳 **Uncle:** Because metadata changes shape depending on the document. An HR policy chunk might have `{ department: \"HR\", doc_type: \"policy\" }`\n\n. A support ticket chunk might have `{ customer_id: 4521, priority: \"high\" }`\n\n. If you tried to make a rigid column for every possible field across every document type, you'd redesign your table every week. `jsonb`\n\ngives you that flexibility — and, this is the part people miss — you can still index *inside* a `jsonb`\n\ncolumn, which we'll do properly in Part 7.\n\n```\nCREATE EXTENSION IF NOT EXISTS vector;\n\nCREATE TABLE document_chunks (\n  id              BIGSERIAL PRIMARY KEY,\n  chunk_text      TEXT NOT NULL,\n  embedding       VECTOR(1536) NOT NULL,\n  embedding_model VARCHAR(50) NOT NULL,\n  content_hash    VARCHAR(64) NOT NULL,\n  metadata        JSONB DEFAULT '{}',\n  created_at      TIMESTAMP DEFAULT now()\n);\n```\n\n👦 **Nephew:** `content_hash`\n\nagain — that's the same SHA-256 idea from Phase 1's file dedup, right?\n\n👨🦳 **Uncle:** Same idea, one level deeper. In Phase 1 we hashed the *whole file* to avoid storing the same file twice. Here we hash each individual *chunk*, to avoid embedding the same chunk twice — because two completely different files can contain the exact same paragraph: a shared boilerplate legal clause, a repeated disclaimer, a standard company intro paragraph copy-pasted into fifty documents. Hold this thought. It becomes real money in Part 10.\n\n👨🦳 **Uncle:** Now, the search speed question. Suppose you have 5 million chunk vectors, each with 1536 dimensions. A user asks a question, you embed it into a query vector. What's the dumbest possible way to find the most similar chunks?\n\n👦 **Nephew:** Compare the query vector against every single one of the 5 million vectors, calculate similarity for each, sort, take the top 5.\n\n👨🦳 **Uncle:** Exactly — that's called a **flat index**, or exhaustive search. Let's see what it actually costs, in real numbers, not vague hand-waving.\n\n```\n5,000,000 vectors × 1536 dimensions each\n\nFor EACH query:\n  Compare against 5,000,000 vectors\n  Each comparison: 1536 multiplications + additions (cosine similarity)\n\nTotal operations per query:\n  5,000,000 × 1536 ≈ 7.68 BILLION operations\n\nOn a typical machine: ~20-30 seconds per query\n```\n\n👦 **Nephew:** 30 seconds?! Nobody waits 30 seconds for a chatbot answer.\n\n👨🦳 **Uncle:** Exactly why brute force only works at small scale — a few thousand vectors, maybe. Past that, you need a fundamentally different strategy: don't check everything, only check the *likely* candidates. This trade-off has a name: **Approximate Nearest Neighbor search**, or **ANN**.\n\n👦 **Nephew:** \"Approximate\"? So it might give the wrong answer?\n\n👨🦳 **Uncle:** It might give you the 4th-closest match instead of the actual 1st-closest, occasionally. That trade is almost always worth it, because the alternative is checking all 5 million vectors exactly, on every single query, forever. You're trading a tiny bit of accuracy for a massive amount of speed.\n\n👦 **Nephew:** Is this like the B-tree index we used on a normal database column, back when we built that email lookup?\n\n👨🦳 **Uncle:** Same *spirit* — avoid scanning everything — but a completely different mechanism, and this distinction trips people up constantly. A B-tree index works because values have a natural sort order: \"find email = x\" can binary-search through sorted order, because \"a\" comes before \"b\" comes before \"c\". But vectors can't be sorted meaningfully in 1536-dimensional space. There's no \"less than\" between two directions in space.\n\nSo vector databases use an entirely different family of tricks. Every real ANN technique, no matter how fancy the name sounds, does one of two things: **groups similar vectors together** so you only search a small group, or **builds a shortcut map** that lets you jump toward the right neighborhood instead of walking through everyone. Let's go through both, properly.\n\n👨🦳 **Uncle:** Remember Google Maps — it doesn't search the whole world to find Bangalore, it narrows down: Country → State → City. IVF works the same way for vectors.\n\n**Step 1 — Training: group vectors into \"neighborhoods\" (clusters)**\n\nBefore any search happens, IVF looks at all your vectors and groups them into K clusters, using an algorithm like k-means. Imagine 5 million vectors get grouped into 1,000 clusters:\n\n| Cluster | Centroid Topic | Vectors |\n|---|---|---|\n| 1 | Programming topics | 5,200 |\n| 2 | HR policy topics | 4,800 |\n| 3 | Financial topics | 5,100 |\n| ... | ... | ... |\n| 1000 | Legal topics | 4,900 |\n\nEach cluster has a **centroid** — the \"average\" vector representing everything in that group.\n\n**Step 2 — Query time: only search the closest clusters**\n\n```\nQuery: \"What is our notice period?\"\n       ↓\nCompare query ONLY against the 1,000 centroids (fast — just 1,000 comparisons)\n       ↓\nFind the 5 closest centroids (say, clusters 3, 47, 112, 289, 501)\n       ↓\nOnly search vectors INSIDE those 5 clusters (~25,000 vectors, not 5 million!)\n       ↓\nReturn top-5 most similar from that much smaller set\n// Conceptual illustration of IVF search (real implementations are C++/Rust,\n// but this shows the logic clearly)\nfunction ivfSearch(queryVector, centroids, clusters, nProbe = 5) {\n  // Step 1: Find nProbe closest centroids (cheap — only 1000 comparisons)\n  const centroidDistances = centroids.map((centroid, i) => ({\n    clusterId: i,\n    distance: cosineSimilarity(queryVector, centroid)\n  }));\n\n  const closestClusters = centroidDistances\n    .sort((a, b) => b.distance - a.distance)\n    .slice(0, nProbe);\n\n  // Step 2: Only search vectors WITHIN those clusters\n  let candidates = [];\n  for (const cluster of closestClusters) {\n    candidates.push(...clusters[cluster.clusterId]);\n  }\n\n  // Step 3: Full precise search, but only on the much smaller candidate set\n  return candidates\n    .map(v => ({ vector: v, score: cosineSimilarity(queryVector, v.embedding) }))\n    .sort((a, b) => b.score - a.score)\n    .slice(0, 5);\n}\n```\n\n👦 **Nephew:** So instead of 5 million comparisons, we do 1,000 (to find clusters) + ~25,000 (inside the clusters) ≈ 26,000. That's roughly 200x fewer operations!\n\n👨🦳 **Uncle:** Exactly the math. And notice the tuning knob — `nProbe`\n\n, how many clusters you search:\n\n| nProbe | Result |\n|---|---|\n| 1 | Fastest, but might miss the real best match (low recall) |\n| 10 | Good balance, catches ~95% of true best matches |\n| 1000 | Same as brute force (searches everything) |\n\nThis is the **recall-vs-speed tradeoff**, and it's the central tension in every vector index that exists.\n\nOne more practical detail: how many clusters (`lists`\n\n, in pgvector's terminology) should you even create in the first place? A common rule of thumb:\n\n```\nlists ≈ sqrt(number of rows)\n\nFor 5 million rows: sqrt(5,000,000) ≈ 2,236, so roughly 2,000-2,500 lists\n\nToo few lists  → clusters are huge, barely faster than brute force.\nToo many lists → clusters are tiny, and centroids get unreliable\n                 without enough data per cluster.\nCREATE INDEX ON document_chunks\nUSING ivfflat (embedding vector_cosine_ops)\nWITH (lists = 2000);\n```\n\n👨🦳 **Uncle:** IVF groups things into neighborhoods. HNSW takes a completely different approach — it builds a multi-level shortcut map, like a highway system.\n\n👦 **Nephew:** Highway system?\n\n👨🦳 **Uncle:** Think about how you'd actually drive from Bangalore to a small village 800 km away. You don't take village roads the whole way.\n\n```\nLayer 2 (Highways):     Bangalore ─────────────────→ Big City X\n                                                          │\nLayer 1 (State roads):  Bangalore ──→ Town A ──→ Town B ──┤\n                                                          │\nLayer 0 (Local roads):  Bangalore ──→ ... ──→ ... ──→ Village\n```\n\n👨🦳 **Uncle:** You take the highway to get *close* fast, then drop to smaller roads only for the final stretch. HNSW builds exactly this structure over your vectors — multiple layers, where the top layer has very few \"long-distance\" connections, and each layer below has progressively more, finer-grained, local connections.\n\n**How HNSW search actually works:**\n\n```\nLayer 2 (few nodes, long connections):\n  Start at entry point → jump to nearest node in this sparse layer\n       ↓ (drop down one layer, using that node as new starting point)\n\nLayer 1 (more nodes, medium connections):\n  From current position → jump to nearest node in this layer\n       ↓ (drop down again)\n\nLayer 0 (ALL nodes, short/local connections):\n  From current position → carefully walk to the actual nearest neighbors\n\nTotal \"hops\" needed: roughly log(N) instead of N\nFor 5 million vectors: ~22 hops instead of 5 million comparisons!\n// Conceptual illustration of HNSW search\nfunction hnswSearch(queryVector, graph, entryPoint, topK = 5) {\n  let currentNode = entryPoint;\n  let currentLayer = graph.topLayer;\n\n  // Phase 1: Greedy descent through upper layers (the \"highway\" phase)\n  while (currentLayer > 0) {\n    currentNode = greedySearchInLayer(queryVector, currentNode, graph, currentLayer);\n    currentLayer--;\n  }\n\n  // Phase 2: Careful search in the bottom layer (the \"local roads\" phase)\n  const candidates = beamSearchInLayer(queryVector, currentNode, graph, 0, topK * 4);\n\n  return candidates\n    .sort((a, b) => b.score - a.score)\n    .slice(0, topK);\n}\n```\n\n👦 **Nephew:** So HNSW is like... a skip list, but for vectors in high-dimensional space?\n\n👨🦳 **Uncle:** That's a genuinely excellent way to think about it. If you've seen skip lists in data structures, HNSW is that exact idea, generalized from a sorted 1-D list to a graph in 1536-dimensional space — sparse at the top for big jumps, dense at the bottom for precision.\n\n| IVF | HNSW | |\n|---|---|---|\n| Search speed | Fast | Usually faster |\n| Build time | Faster to build | Slower to build (constructing the graph) |\n| Memory usage | Lower | Higher (stores graph connections) |\n| Accuracy (recall) | Good, tunable via `nProbe`\n|\nExcellent, tunable via `ef_search`\n|\n| Handling new inserts | Easy — assign to nearest cluster | Harder — inserting into a graph is costlier |\n| Best for | Very large datasets, memory-constrained, heavy insert traffic | Best accuracy/speed balance — the modern default |\n\n👦 **Nephew:** So if HNSW is usually faster and more accurate, why would anyone use IVF?\n\n👨🦳 **Uncle:** Two real reasons. First, memory — at massive scale (hundreds of millions of vectors), HNSW's graph connections take noticeably more RAM than IVF's simpler cluster-list structure. Second, update patterns — if your dataset is constantly getting new vectors (a live chat system logging every message, for instance), IVF handles that more gracefully than rebuilding parts of an HNSW graph. For most RAG systems — a few hundred thousand to a few million chunks, updated periodically rather than constantly — HNSW is the common default today, and it's what most managed vector databases use out of the box.\n\n👦 **Nephew:** Is there anything for when even HNSW's memory usage becomes a problem?\n\n👨🦳 **Uncle:** Yes — **Product Quantization**, or PQ. This doesn't replace IVF or HNSW; it works alongside them, by compressing the vectors themselves.\n\n```\nOriginal vector: 1536 dimensions × 4 bytes each = 6,144 bytes per vector\n\nProduct Quantization:\n  Split the 1536 dimensions into, say, 96 sub-vectors of 16 dimensions each\n  For each sub-vector, find its closest match from a small pre-learned \"codebook\"\n  Store just the CODE (an index into the codebook), not the raw numbers\n\nCompressed vector: 96 codes × 1 byte each = 96 bytes per vector\n\nCompression: 6,144 bytes → 96 bytes = 64x smaller!\n```\n\n👨🦳 **Uncle:** The tradeoff, predictably, is a small loss in precision — you're now comparing compressed approximations, not exact vectors. But for 100 million+ vectors, that memory savings can be the difference between fitting in RAM and not fitting at all. Real large-scale systems often combine all three techniques: IVF to narrow down the neighborhood, PQ to keep everything compact in memory, and a final precise re-ranking step on the small candidate set to recover accuracy.\n\n👦 **Nephew:** We've been saying \"cosine similarity\" this whole time. Is that the only way to measure closeness?\n\n👨🦳 **Uncle:** No — three metrics show up constantly, and picking the wrong one for your use case genuinely hurts results.\n\n| Metric | Measures | Best For | Formula |\n|---|---|---|---|\nCosine Similarity |\nAngle between vectors, ignores magnitude | Text embeddings (most common for RAG) | `(A · B) / (\\ |\nEuclidean Distance (L2) |\nActual straight-line distance | Image embeddings, spatial data | {% raw %}`sqrt(sum((A[i] - B[i])²))`\n|\nDot Product |\nLike cosine, but also factors in magnitude | When magnitude itself is meaningful | `A · B` |\n\n👨🦳 **Uncle:** For text embeddings from OpenAI, Cohere, or most modern embedding models — cosine similarity is the standard, because these models are trained so direction, not magnitude, carries the meaning. But always check your embedding provider's documentation — some models are specifically trained for dot-product comparison instead, and using the wrong metric silently gives you *worse* results, with no error message telling you why.\n\n👨🦳 **Uncle:** Here's a problem vectors alone can never solve, no matter how good your ANN index is. Say your RAG system serves the whole company. HR documents, engineering docs, finance docs — all in one table, all embedded the same way. Someone from Finance asks:\n\n\"What is our expense reimbursement policy?\"\n\n👦 **Nephew:** Pure vector search would search all 5 million chunks — including HR chunks, engineering wiki pages, everything?\n\n👨🦳 **Uncle:** Exactly the problem. Vector similarity alone doesn't know \"this user only has access to Finance documents\" or \"only search documents from the last 6 months.\" That's not a meaning problem — it's a **filtering** problem. This is what metadata indexing solves, and it works completely independently from HNSW or IVF.\n\n👨🦳 **Uncle:** There are two ways to combine \"filter by metadata\" with \"search by vector similarity\" — and picking the wrong one silently breaks your results, without ever throwing an error.\n\n**Post-filtering (the wrong default for most cases):**\n\n```\n1. Run vector search → get top 5 similar chunks (from ALL departments)\n2. THEN filter by department = 'Finance'\n3. Problem: if none of the top 5 happened to be Finance,\n   you get ZERO results — even though relevant Finance\n   chunks exist further down the ranking!\n```\n\n**Pre-filtering (usually correct):**\n\n``` php\n1. FIRST filter: WHERE metadata->>'department' = 'Finance'\n2. THEN run vector search, but ONLY within that filtered set\n3. Correct Finance chunks are guaranteed to be considered\n```\n\n👦 **Nephew:** So we always want pre-filtering?\n\n👨🦳 **Uncle:** Almost always, for access-control and correctness reasons like this. The catch is performance — pre-filtering needs the metadata filter itself to be fast, or you've just recreated the \"scan everything\" problem one step earlier. That's exactly why metadata needs its own index, separate from your HNSW or IVF index on the embedding column.\n\n**Indexing inside jsonb:**\n\n``` php\nCREATE INDEX idx_metadata_department\nON document_chunks\nUSING GIN ((metadata -> 'department'));\n\n-- If you'll filter by department very frequently, extract it into\n-- its own indexed column instead — faster still:\n\nALTER TABLE document_chunks ADD COLUMN department VARCHAR(50)\n  GENERATED ALWAYS AS (metadata->>'department') STORED;\n\nCREATE INDEX idx_department ON document_chunks (department);\n```\n\n**The combined query — the real production pattern:**\n\n``` js\nSELECT\n  chunk_text,\n  metadata,\n  1 - (embedding <=> $1) AS similarity\nFROM document_chunks\nWHERE department = 'Finance'\n  AND created_at > NOW() - INTERVAL '6 months'\nORDER BY embedding <=> $1\nLIMIT 5;\n```\n\n👦 **Nephew:** That `WHERE`\n\nclause runs first, using the metadata index — then the vector similarity ordering happens only on what's left?\n\n👨🦳 **Uncle:** Exactly the mental model. Postgres's query planner uses the metadata index to shrink the candidate set first, then the vector index does its ANN search on that much smaller set.\n\nFilter hard with metadata, search smart with vectors.That sentence is worth remembering more than any specific syntax we've covered today.\n\n👨🦳 **Uncle:** Let's make everything concrete, end to end, since you're already comfortable with Postgres.\n\n```\n-- Enable the extension\nCREATE EXTENSION IF NOT EXISTS vector;\n\n-- Without an index: pgvector does exact brute-force search.\n-- Fine for a few thousand rows, painful past that — exactly Part 2's problem.\n\n-- Create an HNSW index (recommended default for most RAG use cases)\nCREATE INDEX ON document_chunks\nUSING hnsw (embedding vector_cosine_ops)\nWITH (m = 16, ef_construction = 64);\n\n-- m               = how many connections per node (higher = more accurate, more memory)\n-- ef_construction = how thorough the build process is (higher = better index, slower build)\n// Tuning search-time accuracy vs speed for HNSW in pgvector\nawait db.query(`SET hnsw.ef_search = 100;`); // higher = more accurate, slower\n// Default is 40. Push higher when a query needs better recall\n// (say, a compliance-related question), lower for high-volume, low-stakes lookups.\n```\n\n👦 **Nephew:** So `m`\n\nand `ef_construction`\n\naffect how good the index is when it's **built**, and `ef_search`\n\naffects how thorough each individual **query** is?\n\n👨🦳 **Uncle:** Exactly that distinction, and it matters because of *when* you pay for it. You pay the `ef_construction`\n\ncost once, when building the index. You pay the `ef_search`\n\ncost on every single query, forever. Tune the build settings for overall quality; tune the search setting per-query if some queries deserve more accuracy than others.\n\n`vector_cosine_ops`\n\nin that `CREATE INDEX`\n\nline tells Postgres which similarity math to build the index around. pgvector also supports `vector_l2_ops`\n\n(Euclidean) and `vector_ip_ops`\n\n(dot product) — match the index's ops to the metric you actually query with (Part 6), or the index silently won't help at all.\n\n👦 **Nephew:** Pinecone, Qdrant, Weaviate, pgvector, Milvus — how do I actually pick?\n\n👨🦳 **Uncle:** Same instinct as picking a PDF parser back in Phase 1 — match the tool to your actual scale and constraints, don't pick based on hype.\n\n| pgvector | Qdrant | Pinecone | Weaviate | |\n|---|---|---|---|---|\nType |\nPostgres extension | Standalone (self-host or cloud) | Fully managed cloud | Standalone or managed |\nBest for |\nAlready on Postgres, moderate scale (<10M vectors) | Self-hosted control, strong filtering | Zero-ops, massive scale | Hybrid search (keyword + vector) |\nIndex types |\nHNSW, IVFFlat | HNSW | Proprietary (managed) | HNSW |\nCost model |\nFree (just your Postgres bill) | Free self-hosted, paid cloud tier | Pay per vector + query volume | Free self-hosted, paid cloud tier |\nOperational overhead |\nLow — it's just Postgres | Medium — separate service | None — fully managed | Medium — separate service |\n\n👨🦳 **Uncle:** My honest default: if you're already running Postgres and under roughly 5-10 million vectors, pgvector is usually the right starting choice — one less system to operate, and modern pgvector with HNSW is genuinely fast. Reach for a dedicated vector database when you outgrow that, need advanced filtering at real scale, or need to scale vector search independently from your main database.\n\n👦 **Nephew:** Uncle, one more thing bugging me. Embeddings don't go through an LLM chat, so what \"tokens\" are we even saving on this side?\n\n👨🦳 **Uncle:** Two different token costs hide in a RAG pipeline, and most beginners only think about one of them.\n\n| Cost | What it is |\n|---|---|\nEmbedding tokens |\nEvery time you call the embedding API, you pay per token of the text you're embedding — 5M chunks × ~200 tokens each = 1 billion tokens |\nLLM generation tokens |\nEvery retrieved chunk stuffed into the prompt costs tokens again, every single query, forever |\n\n👨🦳 **Uncle:** This is exactly why we stored `content_hash`\n\nback in Part 1. Before calling the embedding API, check first.\n\n``` js\nasync function getOrCreateEmbedding(chunkText, embeddingModel) {\n  const hash = crypto.createHash(\"sha256\").update(chunkText).digest(\"hex\");\n\n  const existing = await db.query(\n    `SELECT embedding FROM document_chunks\n     WHERE content_hash = $1 AND embedding_model = $2`,\n    [hash, embeddingModel]\n  );\n\n  if (existing.rows.length > 0) {\n    console.log(\"Skipped embedding call — already exists\");\n    return existing.rows[0].embedding;\n  }\n\n  const response = await openai.embeddings.create({\n    model: embeddingModel,\n    input: chunkText,\n  });\n\n  return response.data[0].embedding;\n}\n```\n\n👦 **Nephew:** So if the same disclaimer paragraph appears in 200 different uploaded PDFs...\n\n👨🦳 **Uncle:** You call the embedding API exactly once for it, not 200 times. At scale, across thousands of documents with shared boilerplate, this alone can meaningfully cut embedding costs — some real-world document sets have 20-30% content overlap once you actually measure it.\n\n``` js\n// Expensive: 1000 separate API calls, 1000x network overhead\nfor (const chunk of chunks) {\n  await openai.embeddings.create({ model, input: chunk.text });\n}\n\n// Correct: ONE call, same total tokens, far less overhead\nconst response = await openai.embeddings.create({\n  model: \"text-embedding-3-small\",\n  input: chunks.map(c => c.text),   // array of texts\n});\n```\n\n👨🦳 **Uncle:** The token cost is roughly the same either way — you're still paying for the same total text. But batching slashes the *number of requests*, which matters for rate limits, latency, and — if your provider has any per-request overhead — real cost too.\n\n👨🦳 **Uncle:** This connects straight back to Phase 2's Top-K idea. Every chunk you retrieve gets sent to the LLM as context, and you pay generation-side tokens for every single one, on every single query, forever. Retrieving Top-10 \"just to be safe\" instead of Top-5 doesn't just cost double — it also adds noise that can make the LLM's answer *worse*, not better.\n\n👨🦳 **Uncle:** If many users ask near-identical questions — \"what is the notice period\" asked 500 times a month across the company — cache the final answer (or at least the retrieved chunks) keyed by a normalized version of the question, so identical questions don't re-run the whole embed → search → generate pipeline every time.\n\n```\nPHASE 1: DOCUMENT INGESTION ✅\n─────────────────────────────\nPDF Upload → File Hash Check → Parse & Clean → Chunking\n  → Deduplication → Store Chunk Text\n\nPHASE 2: EMBEDDINGS & SEMANTIC SEARCH ✅\n─────────────────────────────\nChunk Text → Tokenization → Embedding Layer → Vector\n  → Cosine Similarity → Top-K Retrieval\n\nTODAY: STORAGE, INDEXING & TOKEN ECONOMICS ← YOU ARE HERE\n─────────────────────────────\nVector + Metadata\n  ↓\nStore in Postgres (pgvector): chunk_text, embedding,\n  content_hash, embedding_model, metadata (jsonb)\n  ↓\nVector Indexing (HNSW / IVF, +PQ at massive scale) → fast, approximate similarity search\n  ↓\nMetadata Indexing (GIN / generated columns) → fast, correct filtering\n  ↓\nCombined Query: filter FIRST (metadata) → search SMART (vectors)\n  ↓\nToken Economics: dedup via content_hash, batch embedding calls,\n  keep Top-K small, cache repeated queries\n\nPHASE 4: QUERY TIME & PRODUCTION SAFETY (next)\n─────────────────────────────\nUser Question → Embed → Pre-filter by metadata → ANN vector search\n  → Top-K chunks → Groundedness checks → LLM → Cited Answer\n```\n\n| Problem | Before Today | After Today |\n|---|---|---|\n| Search accuracy (meaning) | ✅ Solved in Phase 2 | ✅ Unchanged, still relies on it |\n| Search speed at scale | ❌ Checks everything, 30 sec | ✅ ANN index (HNSW/IVF), milliseconds |\n| Memory at massive scale | ❌ Not addressed | ✅ Product Quantization |\n| Access control / filtering | ❌ Not addressed | ✅ Metadata indexing, pre-filtering |\n| Duplicate embedding cost | ❌ Not addressed | ✅ content_hash dedup |\n| Request overhead | ❌ Not addressed | ✅ Batched embedding calls |\n| Repeated identical questions | ❌ Not addressed | ✅ Query-level caching |\n\n`nProbe`\n\n, with `lists ≈ sqrt(row count)`\n\nas a starting rule of thumb`ef_construction`\n\n(build time) and `ef_search`\n\n(query time); the common default for new production systems today👦 **Nephew:** So when someone says \"our vector search is slow,\" the real question isn't \"which tool\" — it's which index, which parameters, how much data, and whether the slowness is even coming from the vector side at all, versus an unindexed metadata filter?\n\n👨🦳 **Uncle:** Exactly the right instinct. \"Vector search\" was never one thing — it's a whole family of tradeoffs between speed, accuracy, memory, and cost, sitting right alongside an equally important filtering problem that has nothing to do with vectors at all. The right answer always depends on your actual scale and constraints, not on which tool sounds most impressive in a job description.\n\n*Next: Query-Time Architecture & Production Safety — wiring pre-filtering and ANN search together, handling embedding model migrations safely, guardrails for when retrieval finds nothing relevant, and rate limiting for a real production RAG API.*\n\n*Remember: less noise, more action. Today is where a demo RAG project turns into a system that survives real traffic, a real access-control policy, and a real bill.*", "url": "https://wpnews.pro/news/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3", "canonical_source": "https://dev.to/surajrkhonde/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3-4n60", "published_at": "2026-07-17 03:02:18+00:00", "updated_at": "2026-07-17 03:28:39.346259+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "developer-tools", "ai-infrastructure"], "entities": ["pgvector", "PostgreSQL", "HNSW", "IVF", "text-embedding-3-small"], "alternates": {"html": "https://wpnews.pro/news/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3", "markdown": "https://wpnews.pro/news/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3.md", "text": "https://wpnews.pro/news/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3.txt", "jsonld": "https://wpnews.pro/news/vector-databases-deep-indexing-token-economics-the-complete-story-phase-3.jsonld"}}