{"slug": "vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is", "title": "Vector search isn't the hard part. Deciding what should be searched is", "summary": "A developer redesigning an AI knowledge system's retrieval pipeline found that vector search alone becomes ineffective as the knowledge base grows, leading to irrelevant chunks and high token costs. They implemented a two-stage architecture with a SQL registry for metadata-based document ranking before vector search, improving retrieval quality more than changing the generation model.", "body_md": "Over the last few weeks I've been redesigning the retrieval pipeline for an AI knowledge system.\n\nInitially, the architecture was fairly typical:\n\nUser Question │ ▼ Vector Search │ ▼ Top K Chunks │ ▼ LLM\n\nIt worked well while the knowledge base was small.\n\nAs more documents were added, I started seeing a few recurring problems:\n\nMore irrelevant chunks being retrieved.\n\nLarger prompts and increasing token costs.\n\nMultiple documents discussing the same topic competing with each other.\n\nVector search returning semantically similar chunks from documents that weren't actually the best source of truth.\n\nI realized the problem wasn't vector search itself.\n\nIt was deciding what should be searched before semantic retrieval even began.\n\nInstead of treating every document equally, I separated the system into two independent stages.\n\nIngestion\n\nDuring document upload, every document is processed once.\n\nThe pipeline extracts structured metadata including:\n\ndocument type\n\nbusiness role\n\ndepartments\n\ntopics\n\nplanner summary\n\nretrieval keywords\n\nauthority score\n\nimportance score\n\nanswerable questions\n\nThat information is stored in a SQL registry, while document chunks and embeddings are stored separately in a vector database.\n\nDocument │ ▼ Metadata Extraction │ ├────────► SQL Registry │ └────────► Chunking + Embeddings │ ▼ Vector Store\n\nQuery Time\n\nInstead of querying the vector database immediately, the retrieval flow became:\n\nUser Question │ ▼ Intent Analysis │ ▼ Registry Ranking │ ▼ Retrieval Planner │ ▼ Selected Documents │ ▼ Vector Search │ ▼ Context Assembly │ ▼ LLM\n\nThe registry acts as a lightweight ranking layer.\n\nRather than searching every document, it produces a ranked candidate set based on signals such as:\n\nauthority\n\nimportance\n\napproval state\n\ndepartments\n\ndocument role\n\nplanner summary\n\nretrieval keywords\n\ntopic overlap\n\nThe planner then decides which documents should actually participate in vector retrieval.\n\nThe vector database never searches the entire workspace anymore.\n\nOnly the planner-selected documents.\n\nA few other changes made a noticeable difference:\n\nsimilarity thresholding before accepting chunks\n\nduplicate chunk removal\n\ntoken budgeting before generation\n\ndynamic chunk limits based on query type\n\nranking retrieved chunks before assembling context\n\nOne interesting observation was that improving retrieval often had a larger impact on answer quality than changing the generation model.\n\nI'm curious whether others have moved beyond \"vector search first\" architectures.\n\nIf you've experimented with retrieval planning, metadata-driven routing, or hybrid retrieval systems, I'd be interested in hearing what worked and what didn't.\n\nComments URL: [https://news.ycombinator.com/item?id=48939470](https://news.ycombinator.com/item?id=48939470)\n\nPoints: 1\n\n# Comments: 0", "url": "https://wpnews.pro/news/vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is", "canonical_source": "https://news.ycombinator.com/item?id=48939470", "published_at": "2026-07-16 19:58:25+00:00", "updated_at": "2026-07-16 20:25:32.598039+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-infrastructure", "ai-tools", "natural-language-processing", "ai-research"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is", "markdown": "https://wpnews.pro/news/vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is.md", "text": "https://wpnews.pro/news/vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is.txt", "jsonld": "https://wpnews.pro/news/vector-search-isn-t-the-hard-part-deciding-what-should-be-searched-is.jsonld"}}