You've seen the tutorials. Spin up Pinecone, call .upsert()
, do a similarity search, ship it. Everyone claps. The demo works.
Then you take it to production and it starts lying to you.
Results that look semantically relevant but aren't. Queries that should match something and return nothing. Latency that makes your users think the app crashed. And the worst part - you don't know why, because the vector database feels like a black box with a fancy API.
This article is about opening that box.
Let's be honest about what "vector database" means, because the term is doing a lot of marketing work right now.
At its core, a vector database is an index optimized for approximate nearest neighbor (ANN) search over high-dimensional float arrays. That's it. The "database" part - persistence, CRUD, filtering, transactions - is infrastructure wrapped around that core capability.
When you store an embedding, you're storing a point in N-dimensional space (typically 768, 1536, or 3072 dimensions depending on your model). When you query, you're asking: "which stored points are closest to this query point, by some distance metric?"
The challenge? Doing exact nearest neighbor search at scale is O(N * D)
- linear in your corpus size times the dimensionality. For a million 1536-dim vectors, that's ~6 billion float comparisons per query. At millisecond latency requirements, that's a hard no.
ANN algorithms trade a small amount of accuracy for massive speed gains. Understanding this trade-off is the first thing most tutorials skip - and it's where production bugs hide.
The algorithm your vector DB uses to build its index determines everything: speed, recall, memory usage, and how it degrades under pressure.
This is what most modern vector DBs use by default (Qdrant, Weaviate, Milvus, pgvector with the right extension). HNSW builds a multi-layer graph where:
Think of it like a highway system. You jump on the highway (top layer), drive toward your destination, exit at the right interchange, and then use local streets (bottom layer) for precision.
Key parameters you need to know:
from qdrant_client.models import VectorParams, Distance
client.create_collection(
collection_name="my_docs",
vectors_config=VectorParams(
size=1536,
distance=Distance.COSINE,
hnsw_config={
"m": 16, # Number of edges per node. Higher = better recall, more memory
"ef_construct": 100, # Construction-time beam width. Higher = better index quality, slower build
}
)
)
results = client.search(
collection_name="my_docs",
query_vector=query_embedding,
limit=10,
search_params={"ef": 128} # Runtime beam width. Higher = better recall, slower query
)
m
and ef_construct
are set at build time and can't change without rebuilding your index. If you're seeing poor recall in production and you set m=4
to save memory, that's your culprit.
Used by FAISS and as an option in pgvector. Divides the vector space into Voronoi cells (clusters), assigns vectors to their nearest centroid, then searches only a subset of cells at query time.
import faiss
import numpy as np
dimension = 1536
n_clusters = 1024 # Number of Voronoi cells
quantizer = faiss.IndexFlatL2(dimension)
index = faiss.IndexIVFFlat(quantizer, dimension, n_clusters)
index.train(training_vectors) # Needs representative data
index.add(corpus_vectors)
index.nprobe = 32
distances, indices = index.search(query_vector, k=10)
IVF gotcha: the cluster centroids are learned during training. If your data distribution shifts significantly (new document types, different topics), your centroid structure becomes suboptimal and recall tanks. You don't get an error. You just quietly get worse results.
Most people use cosine similarity because the tutorial said so. Here's when that's wrong.
| Metric | Formula | Use When |
|---|---|---|
| Cosine | 1 - (A·B / ‖A‖‖B‖) |
|
| Direction matters, magnitude doesn't. Good for normalized text embeddings | ||
| Dot Product | -(A·B) |
|
| Embeddings are already normalized (OpenAI's are). Faster than cosine | ||
| Euclidean (L2) | ‖A-B‖ |
|
| Magnitude carries meaning. Image embeddings, some multimodal models |
OpenAI's text-embedding-3-*
embeddings are normalized to unit length. Cosine similarity on unit vectors is mathematically equivalent to dot product. Using cosine adds a normalization step that's pure overhead.
VectorParams(size=1536, distance=Distance.DOT)
SELECT content, embedding <#> query_embedding AS score
FROM documents
ORDER BY score
LIMIT 10;
The difference in latency is small at low scale. At 10M+ vectors, it's measurable.
Here's a thing that will haunt you: your ANN search does not always return the true nearest neighbors.
It returns approximate nearest neighbors. That's the A in ANN. By definition, you may miss results that should have ranked in your top-K.
How bad is it? It depends on your index config and your data. You can measure it:
import numpy as np
from qdrant_client import QdrantClient
def measure_recall(client, collection_name, test_queries, ground_truth_ids, k=10):
"""
Compare ANN results against brute-force exact search.
ground_truth_ids: list of lists, true top-k ids per query
"""
hits = 0
total = len(test_queries) * k
for query, true_ids in zip(test_queries, ground_truth_ids):
ann_results = client.search(
collection_name=collection_name,
query_vector=query,
limit=k
)
ann_ids = {r.id for r in ann_results}
hits += len(ann_ids & set(true_ids))
return hits / total # recall@k
Production target: ≥ 0.95 recall@10. Anything below that and your RAG pipeline is silently missing relevant context before GPT-4 ever sees it.
Pure vector search has a well-known failure mode: it doesn't handle rare terms well.
If your corpus contains "RFC 7807 Problem Details" or a specific error code like E_INVALIDARG_0x80070057
, embedding similarity will dilute the match across semantically adjacent concepts. A user querying for the exact string gets mushy results.
The solution is hybrid search: combine dense vector search with sparse BM25-style keyword search, then fuse the rankings.
from qdrant_client import QdrantClient
from qdrant_client.models import (
SparseVectorParams, VectorParams,
SparseIndexParams, Distance, NamedVector, NamedSparseVector
)
client.create_collection(
collection_name="hybrid_docs",
vectors_config={
"dense": VectorParams(size=1536, distance=Distance.COSINE),
},
sparse_vectors_config={
"sparse": SparseVectorParams(index=SparseIndexParams(on_disk=False))
}
)
from fastembed import SparseTextEmbedding, TextEmbedding
dense_model = TextEmbedding("BAAI/bge-small-en-v1.5")
sparse_model = SparseTextEmbedding("prithivida/Splade_PP_en_v1")
text = "RFC 7807 Problem Details for HTTP APIs"
dense_vec = list(dense_model.embed([text]))[0]
sparse_vec = list(sparse_model.embed([text]))[0]
from qdrant_client.models import Prefetch, FusionQuery, Fusion
results = client.query_points(
collection_name="hybrid_docs",
prefetch=[
Prefetch(query=dense_vec.tolist(), using="dense", limit=20),
Prefetch(
query=SparseVector(indices=sparse_vec.indices.tolist(),
values=sparse_vec.values.tolist()),
using="sparse", limit=20
),
],
query=FusionQuery(fusion=Fusion.RRF),
limit=10
)
RRF (Reciprocal Rank Fusion) combines the rank lists without needing score normalization. The formula is simple:
RRF_score(d) = Σ 1 / (k + rank_i(d))
Where k
is a constant (usually 60) and rank_i(d)
is the document's rank in each result list. Documents appearing in both lists get a significant boost.
Hybrid search consistently outperforms pure dense search on real-world corpora by 5–15% on NDCG@10 - especially for domain-specific or technical content.
Vector DBs let you pre-filter by metadata before (or after) the ANN search. This sounds simple. It's actually one of the most common performance footguns.
Pre-filtering (filter before ANN): Apply your metadata filter first, reduce the candidate set, then run ANN on the smaller set.
Problem: if your filter is very selective (e.g., user_id = "abc123"
in a multi-tenant system), the candidate set might be tiny. HNSW graph navigation assumes a large, connected graph. A sparse subgraph destroys recall.
Post-filtering (ANN then filter): Run ANN on the full corpus, retrieve top-N, then apply filter. You need to over-fetch significantly to compensate for filtered-out results.
client.create_payload_index(
collection_name="my_docs",
field_name="tenant_id",
field_schema="keyword" # or "integer", "float", "geo"
)
results = client.search(
collection_name="my_docs",
query_vector=query_embedding,
query_filter=Filter(
must=[FieldCondition(key="tenant_id", match=MatchValue(value="abc123"))]
),
limit=10
)
Rule of thumb: if your filter reduces the corpus below ~1000 vectors, you're effectively doing brute-force search. That's fine - just know it and set expectations accordingly.
This isn't vector DB internals, but it's so deeply related that skipping it would be malpractice.
Your retrieval quality is bounded by your chunking quality. The vector DB can only return what you gave it.
Most tutorials show:
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
chunks = text_splitter.split_text(document)
The problems:
Better: semantic chunking
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
splitter = SemanticChunker(
OpenAIEmbeddings(),
breakpoint_threshold_type="percentile",
breakpoint_threshold_amount=95 # Split when semantic shift exceeds 95th percentile
)
chunks = splitter.split_text(document)
This embeds sentences, calculates cosine distance between adjacent sentence pairs, and splits at significant semantic shifts.
Even better: store both chunk and parent document
from langchain.retrievers import ParentDocumentRetriever
from langchain.storage import InMemoryStore
child_splitter = RecursiveCharacterTextSplitter(chunk_size=200)
parent_splitter = RecursiveCharacterTextSplitter(chunk_size=2000)
retriever = ParentDocumentRetriever(
vectorstore=vectorstore,
docstore=InMemoryStore(),
child_splitter=child_splitter,
parent_splitter=parent_splitter,
)
Small chunks match with high precision. The returned context is the larger parent - so your LLM gets enough surrounding information to reason correctly.
If you're not measuring this stuff, you're flying blind:
import time
from dataclasses import dataclass
from typing import Optional
@dataclass
class RetrievalTrace:
query: str
query_embedding_ms: float
search_ms: float
num_results: int
top_score: float
bottom_score: float
score_spread: float # top - bottom; low spread = retrieval is uncertain
filter_applied: Optional[dict]
collection_name: str
def traced_search(client, collection_name, query_text, embed_fn, k=5, filter=None):
t0 = time.perf_counter()
embedding = embed_fn(query_text)
embed_ms = (time.perf_counter() - t0) * 1000
t1 = time.perf_counter()
results = client.search(
collection_name=collection_name,
query_vector=embedding,
limit=k,
query_filter=filter
)
search_ms = (time.perf_counter() - t1) * 1000
scores = [r.score for r in results]
trace = RetrievalTrace(
query=query_text,
query_embedding_ms=embed_ms,
search_ms=search_ms,
num_results=len(results),
top_score=scores[0] if scores else 0,
bottom_score=scores[-1] if scores else 0,
score_spread=(scores[0] - scores[-1]) if len(scores) > 1 else 0,
filter_applied=filter,
collection_name=collection_name
)
log_trace(trace)
return results
What to watch:
score_spread
near 0 means all results look equally similar - the query probably didn't match anything welltop_score
below your threshold (tune per model, but ~0.75 for cosine is a reasonable starting floor) means you're returning noiseQuick opinionated guide for 2026:
| Scenario | Recommendation |
|---|---|
| Prototype / hobby | ChromaDB (in-process, zero infra) |
| Production, self-hosted | Qdrant (best performance, Rust core, Docker-native) |
| Already on Postgres | pgvector + pgvectorscale |
| Enterprise, managed | Pinecone or Weaviate Cloud |
| Need multimodal (text + image) | Weaviate or Milvus |
| Massive scale (100M+ vectors) | Milvus or Pinecone |
Don't use a vector DB for everything. If your corpus is under ~10,000 documents, cosine search over an in-memory numpy array with np.dot
is fast enough and eliminates an entire infrastructure dependency.
import numpy as np
corpus_embeddings = np.load("embeddings.npy") # shape: (N, 1536)
query_embedding = np.array(embed(query)) # shape: (1536,)
scores = corpus_embeddings @ query_embedding
top_k_indices = np.argsort(scores)[::-1][:10]
No database. No network calls. No ops burden. Just math.
Pull all of this together and you get a mental model for diagnosing RAG failures:
ef
/nprobe
Vector databases are not magic retrieval oracles. They're approximate spatial indexes with a product wrapper. Once you understand the approximation, the trade-offs, and the failure modes - you can actually build reliable systems with them.
If this was useful, I write about Python backend and AI engineering on dev.to. The good stuff is in the details.