# My RAG retrieval quality is tanking as my dataset grows

> Source: <https://promptcube3.com/en/threads/6615/>
> Published: 2026-08-16 22:49:24+00:00

# My RAG retrieval quality is tanking as my dataset grows

I've been building a

Since I'm using

I'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?

I'm using

Right now, I store metadata in a separate Python list (

[RAG](/en/tags/rag/)pipeline using`SentenceTransformer`

and FAISS. On a tiny test set, everything looks great, but as I scale up, I'm hitting a wall with retrieval precision. I'm currently using `IndexFlatIP`

with L2-normalized vectors to simulate cosine similarity. It's basically a brute-force search, which is fine for a few hundred documents, but I'm worried about the latency and accuracy trade-offs as I move toward a real-world deployment.Here is the core of my current implementation:

``` python
class VectorStore:

 def __init__(self):
 self.model = SentenceTransformer(EMBEDDING_MODEL)
 self.index = None
 self.chunks = []

 def build(self, chunks):

 self.chunks = chunks

 texts = [chunk["text"] for chunk in chunks]

 embeddings = self.model.encode(
 texts,
 convert_to_numpy=True,
 show_progress_bar=True
 ).astype("float32")

 # Normalize embeddings so inner product = cosine similarity
 faiss.normalize_L2(embeddings)

 dimension = embeddings.shape[1]

 self.index = faiss.IndexFlatIP(dimension)
 self.index.add(embeddings)

 def search(self, query, k=5):

 query_embedding = self.model.encode(
 [query],
 convert_to_numpy=True
 ).astype("float32")

 faiss.normalize_L2(query_embedding)

 scores, indices = self.index.search(
 query_embedding,
 k
 )

 results = []

 for score, index in zip(scores[0], indices[0]):
 if index == -1:
 continue

 results.append({
 "text": self.chunks[index]["text"],
 "chunk_id": self.chunks[index]["chunk_id"],
 "score": float(score)
 })

 return results
```

I've realized that just having "similar" vectors isn't enough for a production-grade AI workflow. I have a few specific technical hurdles I'm trying to clear:

## The Indexing Bottleneck

Since I'm using

`IndexFlatIP`

, every query scans the entire index. I'm considering switching to `IndexHNSWFlat`

or `IndexIVFFlat`

to speed things up, but I'm unsure about the recall hit. In a practical tutorial for RAG, you often see HNSW recommended for speed, but is the memory overhead worth it compared to something like `IndexIVFPQ`

for larger datasets?## The "Vector Search is Not Enough" Problem

I'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?

## Embedding Model Selection

I'm using

`all-MiniLM-L6-v2`

because it's fast, but I suspect it's the weak link. Should I be focusing on Recall@K metrics when picking a new model, or does the choice of embedding model actually impact how the FAISS index behaves?## Metadata Filtering Struggles

Right now, I store metadata in a separate Python list (

`self.chunks`

) and map it back via indices. This makes filtering (e.g., "only search within document_X") a nightmare because I have to filter *after*the search, which ruins my Top-K results. I need a better way to handle filtered vector search without iterating through the entire result set manually.[Next My CMAKE_BUILD_TYPE checks are failing on Windows with MSVC →](/en/threads/6601/)

Free AI toolbox — all free to use

Step-by-step guides and pitfalls for this path are in

[an AI side-hustle playbook](https://tanyan888.com/), with plenty of directly applicable cases.## All Replies （4）

S

K

Welcome to the club. Mine started hallucinating wild stuff the second I hit 1k docs.

0

J

Are you using a specific chunking strategy or just splitting by character count?

0