Close enough isn’t good enough: teaching an AI assistant to find the exact answer

A developer building a retrieval-augmented generation (RAG) assistant for document processing replaced a split SQLite-plus-Chroma store with a single Postgres database using a pgvector column and a generated tsvector column, so deleting a document and its chunks now takes one statement via ON DELETE CASCADE. The developer found that Postgres's websearch_to_tsquery returned nothing for most natural-language questions because it ANDs every term and parses codes like ERR-4021 as the word "err" plus "-4021"; routing the question through the same to_tsvector('english', ...) call used for documents and joining the lexemes with OR, ranked by ts_rank_cd, made the ERR-4021 chunk score 0.2 versus 0.1 for ERR-4012. The developer then combined vector and keyword results with Reciprocal Rank Fusion to keep the system free to run.

My last article on Doc Processing AI ended with a “what’s next” list: move to pgvector, try hybrid retrieval, and grow the golden set into something meaningful. This is that follow-up. The motivation is a failure every pure vector search eventually hits. Ask an incident runbook “What should on-call do about ERR-4021?” and the embedding of that question sits almost exactly as close to the ERR-4012 section as to the ERR-4021 one. To an embedding model, two error codes that differ by two digits mean nearly the same thing. To the engineer on call at 3am, they’re different problems with different fixes. Keyword search gets this right for free, but it’s bad at paraphrases. Production RAG systems usually run both and combine the results, then rerank. I wanted to build that properly, measure whether it actually helps, and keep the whole thing free to run. The design goals were: The first version stored document metadata in SQLite and vectors in Chroma. That works until you need both at once: filtering by document, deleting a document and its vectors together, or joining a search hit back to its filename. Every one of those crossed two stores. Now a chunk is just a row. The embedding is a pgvector column and the full-text index is a generated column, so Postgres keeps it in sync on every insert: class Chunk Base : tablename = "chunks" id: Mapped str = mapped column String 64 , primary key=True document id: Mapped str = mapped column String 36 , ForeignKey "documents.id", ondelete="CASCADE" , nullable=False, index=True chunk index: Mapped int = mapped column Integer, nullable=False content: Mapped str = mapped column Text, nullable=False embedding = mapped column Vector get settings .embed dim , nullable=False 768 tsv = mapped column TSVECTOR, Computed "to tsvector 'english', content ", persisted=True table args = Index "ix chunks tsv", "tsv", postgresql using="gin" , Index "ix chunks embedding", "embedding", postgresql using="hnsw", postgresql ops={"embedding": "vector cosine ops"} , Deleting a document is now one statement. ON DELETE CASCADE removes its chunks, which used to need a separate call to Chroma. My first keyword search used websearch to tsquery, the function most Postgres full-text examples reach for. It found nothing for most questions. The psql output shows why: Two things go wrong. First, websearch to tsquery ANDs every term, so a full natural-language question only matches a chunk that contains all of its words. That's fine for a search box, but questions aren't search-box queries. Second, Postgres's parser treats ERR-4021 as the word err followed by the number -4021, minus sign included. A query I built in Python by splitting on punctuation sent 4021, which never matches -4021. The fix was to stop tokenizing in Python at all. The question goes through the same to tsvector 'english', ... call as the documents, and the resulting lexemes are joined with OR. ts rank cd then ranks chunks that share more of them higher, which behaves a lot like BM25: python def or tsquery question: str : lexemes = func.unnest func.to tsvector "english", question .table valued "lexeme", "positions", "weights" return select func.to tsquery "simple", func.coalesce func.string agg func.quote literal column "lexeme" , " | " , "" , .select from lexemes .scalar subquery Now the ERR-4021 chunk scores 0.2 and ERR-4012 scores 0.1, because only one of them contains -4021. Product codes like NV-220X work the same way. A question made only of stopwords produces an empty query and matches nothing, which is the behavior you want. Vector search returns cosine similarities between 0 and 1. Keyword search returns ts rank cd scores, which are on a completely different scale. Normalizing and adding them is fiddly and fragile. Reciprocal Rank Fusion avoids the problem by ignoring scores entirely and using only rank. Each chunk gets 1 / k + rank from every list it appears in, with k = 60: php def reciprocal rank fusion result lists: list list Candidate , k: int = 60 - list Candidate : fused: dict str, Candidate = {} for results in result lists: for rank, candidate in enumerate results, start=1 : merged = fused.get candidate.chunk id if merged is None: merged = Candidate chunk id=candidate.chunk id, document id=candidate.document id, filename=candidate.filename, chunk index=candidate.chunk index, content=candidate.content, scores={"rrf": 0.0}, fused candidate.chunk id = merged merged.scores.update candidate.scores merged.scores "rrf" += 1.0 / k + rank return sorted fused.values , key=lambda c: c.scores "rrf" , reverse=True A chunk that ranks well in both lists beats one that’s first in only one. Both searches run concurrently with asyncio.gather, each in its own database session, and each returns 20 candidates for the fusion step. Both searches score the question and a chunk separately. A cross-encoder reads them together and outputs a single relevance score, which is much more accurate but too slow to run over a whole corpus. So it only runs over the fused candidates. I used ms-marco-MiniLM-L-6-v2. It's small about 90 MB , runs on a CPU in milliseconds per pair, and costs nothing. Two details mattered in an async API: The model is an optional install pip install -e ". rerank " , because it pulls in PyTorch. Without it you can still run RETRIEVAL MODE=hybrid. The original golden set had 8 questions over 3 documents, and every mode already scored 100% on it. It couldn’t tell the modes apart. So I added 5 documents designed to be hard, and grew the set to 22 questions: I also changed what counts as a hit. “The right file was retrieved” is too generous when the 2025 and 2026 policies sit in different files but the answer depends on picking the right one. Each case now has an evidence string, and a hit means the retrieved chunk contains it. The runner reports hit@1, hit@4, and MRR mean reciprocal rank per mode: Hybrid search alone lifts hit@1 from 73% to 86%, and the reranker takes it to 91%. MRR goes from 0.864 to 0.936. The gains are exactly where I expected them: It’s not all wins. Two cases got worse with the reranker: damage-report fell from 1 to 3, and travel-2026-per-diem from 2 to 4. MiniLM is a small model trained on web search queries, and it has blind spots. A bigger reranker such as bge-reranker-base is one setting away, and the eval will show whether it's worth the extra latency. For answers, each case is also scored on whether the response contains the expected fact and whether an LLM judge finds it faithful to the retrieved context. With hybrid + rerank retrieval, all 22 passed both checks. The free tier was the most practical constraint of the project. Gemini’s free tier allows 5 chat requests per minute and only 20 per day on gemini-3.6-flash. One answer-eval run needs 44 requests: an answer and a judge call per case. My first run died at case 8 on the per-minute limit, and the second ran out of the daily quota. Three changes made it workable: Retrieval evals are effectively free: 22 questions need 22 embedding calls and no chat calls. That’s the part I can run on every change. I didn’t want to mock the database. The interesting bugs here tokenization, index types, cascade deletes only show up in real Postgres. The test suite creates a separate rag test database, builds the schema, and inserts chunks with synthetic embeddings: hand-made vectors that point along chosen axes, so a test can control exactly which chunk is "closest" without calling Gemini. The key test builds the exact failure from the intro. The query vector points at ERR-4012, but the question names ERR-4021. Vector search alone misses it, hybrid search brings it into the top results, and the reranker puts it first. One test failure taught me something about RRF. My first fixture had only two runbook chunks. ERR-4012 was 1 by vector and 2 by keyword, and ERR-4021 was the reverse, so both got exactly the same RRF score and the winner came down to sort order. That isn’t a bug in RRF. It’s what RRF should do when the two signals disagree evenly, and it’s the reason for having a reranker at the end. I rewrote the fixture to reflect the real scenario, where vector search ranks the right chunk low rather than a close second. I built this with an AI coding assistant Claude Code doing much of the implementation, while I made the decisions: Postgres instead of a dedicated vector database, a local reranker instead of a paid API, chunk-level evidence instead of file-level hits, and a lite model for evals rather than paying for quota. The assistant wrote code and tests, ran them, and reported failures back. The tokenization bug, the asyncpg codec conflict, and the RRF tie all came out of that loop of running real tests and reading real output. The corpus is still small: 8 documents and 17 chunks. Every mode finds the evidence somewhere in its top 4, so hit@4 is 100% across the board, and only hit@1 and MRR separate them. The React UI didn’t change in this round. And 22 out of 22 on answers says more about how easy the questions are than about the model. Next up: The code is on GitHub: github.com/Eric-Souza/gemini-rag-doc-processing https://github.com/Eric-Souza/gemini-rag-doc-processing . Run docker compose up -d postgres, add a free Gemini key, and python -m evals.run evals --seed --retrieval-only will reproduce the comparison table above. If you’ve tuned hybrid search in production, I’d like to hear what worked for you, especially which reranker you ended up with. Leave a comment. Close enough isn’t good enough: teaching an AI assistant to find the exact answer https://blog.devgenius.io/close-enough-isnt-good-enough-teaching-an-ai-assistant-to-find-the-exact-answer-1d127caad4b7 was originally published in Dev Genius https://blog.devgenius.io on Medium, where people are continuing the conversation by highlighting and responding to this story.