Vector search isn't the hard part. Deciding what should be searched is

A developer redesigning an AI knowledge system's retrieval pipeline found that vector search alone becomes ineffective as the knowledge base grows, leading to irrelevant chunks and high token costs. They implemented a two-stage architecture with a SQL registry for metadata-based document ranking before vector search, improving retrieval quality more than changing the generation model.

Over the last few weeks I've been redesigning the retrieval pipeline for an AI knowledge system. Initially, the architecture was fairly typical: User Question │ ▼ Vector Search │ ▼ Top K Chunks │ ▼ LLM It worked well while the knowledge base was small. As more documents were added, I started seeing a few recurring problems: More irrelevant chunks being retrieved. Larger prompts and increasing token costs. Multiple documents discussing the same topic competing with each other. Vector search returning semantically similar chunks from documents that weren't actually the best source of truth. I realized the problem wasn't vector search itself. It was deciding what should be searched before semantic retrieval even began. Instead of treating every document equally, I separated the system into two independent stages. Ingestion During document upload, every document is processed once. The pipeline extracts structured metadata including: document type business role departments topics planner summary retrieval keywords authority score importance score answerable questions That information is stored in a SQL registry, while document chunks and embeddings are stored separately in a vector database. Document │ ▼ Metadata Extraction │ ├────────► SQL Registry │ └────────► Chunking + Embeddings │ ▼ Vector Store Query Time Instead of querying the vector database immediately, the retrieval flow became: User Question │ ▼ Intent Analysis │ ▼ Registry Ranking │ ▼ Retrieval Planner │ ▼ Selected Documents │ ▼ Vector Search │ ▼ Context Assembly │ ▼ LLM The registry acts as a lightweight ranking layer. Rather than searching every document, it produces a ranked candidate set based on signals such as: authority importance approval state departments document role planner summary retrieval keywords topic overlap The planner then decides which documents should actually participate in vector retrieval. The vector database never searches the entire workspace anymore. Only the planner-selected documents. A few other changes made a noticeable difference: similarity thresholding before accepting chunks duplicate chunk removal token budgeting before generation dynamic chunk limits based on query type ranking retrieved chunks before assembling context One interesting observation was that improving retrieval often had a larger impact on answer quality than changing the generation model. I'm curious whether others have moved beyond "vector search first" architectures. If you've experimented with retrieval planning, metadata-driven routing, or hybrid retrieval systems, I'd be interested in hearing what worked and what didn't. Comments URL: https://news.ycombinator.com/item?id=48939470 https://news.ycombinator.com/item?id=48939470 Points: 1 Comments: 0