{"slug": "my-rag-retrieval-quality-is-tanking-as-my-dataset-grows", "title": "My RAG retrieval quality is tanking as my dataset grows", "summary": "A developer reports that retrieval precision in their RAG pipeline degrades as the dataset grows, using SentenceTransformer's all-MiniLM-L6-v2 and FAISS IndexFlatIP with L2-normalized vectors. They are considering switching to HNSW or IVF indexes, adding a Cross-Encoder reranker, and improving metadata filtering to address the issue.", "body_md": "# My RAG retrieval quality is tanking as my dataset grows\n\nI've been building a\n\nSince I'm using\n\nI'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?\n\nI'm using\n\nRight now, I store metadata in a separate Python list (\n\n[RAG](/en/tags/rag/)pipeline using`SentenceTransformer`\n\nand FAISS. On a tiny test set, everything looks great, but as I scale up, I'm hitting a wall with retrieval precision. I'm currently using `IndexFlatIP`\n\nwith L2-normalized vectors to simulate cosine similarity. It's basically a brute-force search, which is fine for a few hundred documents, but I'm worried about the latency and accuracy trade-offs as I move toward a real-world deployment.Here is the core of my current implementation:\n\n``` python\nclass VectorStore:\n\n def __init__(self):\n self.model = SentenceTransformer(EMBEDDING_MODEL)\n self.index = None\n self.chunks = []\n\n def build(self, chunks):\n\n self.chunks = chunks\n\n texts = [chunk[\"text\"] for chunk in chunks]\n\n embeddings = self.model.encode(\n texts,\n convert_to_numpy=True,\n show_progress_bar=True\n ).astype(\"float32\")\n\n # Normalize embeddings so inner product = cosine similarity\n faiss.normalize_L2(embeddings)\n\n dimension = embeddings.shape[1]\n\n self.index = faiss.IndexFlatIP(dimension)\n self.index.add(embeddings)\n\n def search(self, query, k=5):\n\n query_embedding = self.model.encode(\n [query],\n convert_to_numpy=True\n ).astype(\"float32\")\n\n faiss.normalize_L2(query_embedding)\n\n scores, indices = self.index.search(\n query_embedding,\n k\n )\n\n results = []\n\n for score, index in zip(scores[0], indices[0]):\n if index == -1:\n continue\n\n results.append({\n \"text\": self.chunks[index][\"text\"],\n \"chunk_id\": self.chunks[index][\"chunk_id\"],\n \"score\": float(score)\n })\n\n return results\n```\n\nI've realized that just having \"similar\" vectors isn't enough for a production-grade AI workflow. I have a few specific technical hurdles I'm trying to clear:\n\n## The Indexing Bottleneck\n\nSince I'm using\n\n`IndexFlatIP`\n\n, every query scans the entire index. I'm considering switching to `IndexHNSWFlat`\n\nor `IndexIVFFlat`\n\nto speed things up, but I'm unsure about the recall hit. In a practical tutorial for RAG, you often see HNSW recommended for speed, but is the memory overhead worth it compared to something like `IndexIVFPQ`\n\nfor larger datasets?## The \"Vector Search is Not Enough\" Problem\n\nI'm noticing that cosine similarity often retrieves chunks that are semantically related but logically useless for the actual answer. I'm thinking about adding a reranker stage—pulling 50 candidates via FAISS and then narrowing it down to the top 5 using a Cross-Encoder. Does this actually solve the precision issue, or is it just adding latency to the pipeline?\n\n## Embedding Model Selection\n\nI'm using\n\n`all-MiniLM-L6-v2`\n\nbecause it's fast, but I suspect it's the weak link. Should I be focusing on Recall@K metrics when picking a new model, or does the choice of embedding model actually impact how the FAISS index behaves?## Metadata Filtering Struggles\n\nRight now, I store metadata in a separate Python list (\n\n`self.chunks`\n\n) and map it back via indices. This makes filtering (e.g., \"only search within document_X\") a nightmare because I have to filter *after*the search, which ruins my Top-K results. I need a better way to handle filtered vector search without iterating through the entire result set manually.[Next My CMAKE_BUILD_TYPE checks are failing on Windows with MSVC →](/en/threads/6601/)\n\nFree AI toolbox — all free to use\n\nStep-by-step guides and pitfalls for this path are in\n\n[an AI side-hustle playbook](https://tanyan888.com/), with plenty of directly applicable cases.## All Replies （4）\n\nS\n\nK\n\nWelcome to the club. Mine started hallucinating wild stuff the second I hit 1k docs.\n\n0\n\nJ\n\nAre you using a specific chunking strategy or just splitting by character count?\n\n0", "url": "https://wpnews.pro/news/my-rag-retrieval-quality-is-tanking-as-my-dataset-grows", "canonical_source": "https://promptcube3.com/en/threads/6615/", "published_at": "2026-08-16 22:49:24+00:00", "updated_at": "2026-08-16 23:11:39.647410+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "ai-infrastructure", "ai-tools"], "entities": ["SentenceTransformer", "FAISS", "IndexFlatIP", "IndexHNSWFlat", "IndexIVFFlat", "IndexIVFPQ", "all-MiniLM-L6-v2", "Cross-Encoder"], "alternates": {"html": "https://wpnews.pro/news/my-rag-retrieval-quality-is-tanking-as-my-dataset-grows", "markdown": "https://wpnews.pro/news/my-rag-retrieval-quality-is-tanking-as-my-dataset-grows.md", "text": "https://wpnews.pro/news/my-rag-retrieval-quality-is-tanking-as-my-dataset-grows.txt", "jsonld": "https://wpnews.pro/news/my-rag-retrieval-quality-is-tanking-as-my-dataset-grows.jsonld"}}