I've been building a
Since I'm using
I'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?
I'm using
Right now, I store metadata in a separate Python list (
RAGpipeline usingSentenceTransformer
and FAISS. On a tiny test set, everything looks great, but as I scale up, I'm hitting a wall with retrieval precision. I'm currently using IndexFlatIP
with L2-normalized vectors to simulate cosine similarity. It's basically a brute-force search, which is fine for a few hundred documents, but I'm worried about the latency and accuracy trade-offs as I move toward a real-world deployment.Here is the core of my current implementation:
class VectorStore:
def __init__(self):
self.model = SentenceTransformer(EMBEDDING_MODEL)
self.index = None
self.chunks = []
def build(self, chunks):
self.chunks = chunks
texts = [chunk["text"] for chunk in chunks]
embeddings = self.model.encode(
texts,
convert_to_numpy=True,
show_progress_bar=True
).astype("float32")
faiss.normalize_L2(embeddings)
dimension = embeddings.shape[1]
self.index = faiss.IndexFlatIP(dimension)
self.index.add(embeddings)
def search(self, query, k=5):
query_embedding = self.model.encode(
[query],
convert_to_numpy=True
).astype("float32")
faiss.normalize_L2(query_embedding)
scores, indices = self.index.search(
query_embedding,
k
)
results = []
for score, index in zip(scores[0], indices[0]):
if index == -1:
continue
results.append({
"text": self.chunks[index]["text"],
"chunk_id": self.chunks[index]["chunk_id"],
"score": float(score)
})
return results
I've realized that just having "similar" vectors isn't enough for a production-grade AI workflow. I have a few specific technical hurdles I'm trying to clear:
The Indexing Bottleneck #
Since I'm using
IndexFlatIP
, every query scans the entire index. I'm considering switching to IndexHNSWFlat
or IndexIVFFlat
to speed things up, but I'm unsure about the recall hit. In a practical tutorial for RAG, you often see HNSW recommended for speed, but is the memory overhead worth it compared to something like IndexIVFPQ
for larger datasets?## The "Vector Search is Not Enough" Problem
I'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?
Embedding Model Selection #
I'm using
all-MiniLM-L6-v2
because it's fast, but I suspect it's the weak link. Should I be focusing on Recall@K metrics when picking a new model, or does the choice of embedding model actually impact how the FAISS index behaves?## Metadata Filtering Struggles
Right now, I store metadata in a separate Python list (
self.chunks
) and map it back via indices. This makes filtering (e.g., "only search within document_X") a nightmare because I have to filter afterthe search, which ruins my Top-K results. I need a better way to handle filtered vector search without iterating through the entire result set manually.Next My CMAKE_BUILD_TYPE checks are failing on Windows with MSVC →
Free AI toolbox — all free to use
Step-by-step guides and pitfalls for this path are in
an AI side-hustle playbook, with plenty of directly applicable cases.## All Replies （4）
S
K
Welcome to the club. Mine started hallucinating wild stuff the second I hit 1k docs.
0
J
Are you using a specific chunking strategy or just splitting by character count?
0