# Close enough isn’t good enough: teaching an AI assistant to find the exact answer > Source: > Published: 2026-10-31 06:03:40+00:00 My last article on **Doc Processing AI** ended with a “what’s next” list: move to pgvector, try hybrid retrieval, and grow the golden set into something meaningful. This is that follow-up. The motivation is a failure every pure vector search eventually hits. Ask an incident runbook “What should on-call do about ERR-4021?” and the embedding of that question sits almost exactly as close to the ERR-4012 section as to the ERR-4021 one. To an embedding model, two error codes that differ by two digits mean nearly the same thing. To the engineer on call at 3am, they’re different problems with different fixes. Keyword search gets this right for free, but it’s bad at paraphrases. Production RAG systems usually run both and combine the results, then rerank. I wanted to build that properly, measure whether it actually helps, and keep the whole thing free to run. The design goals were: The first version stored document metadata in SQLite and vectors in Chroma. That works until you need both at once: filtering by document, deleting a document and its vectors together, or joining a search hit back to its filename. Every one of those crossed two stores. Now a chunk is just a row. The embedding is a pgvector column and the full-text index is a generated column, so Postgres keeps it in sync on every insert: ``` class Chunk(Base): __tablename__ = "chunks" id: Mapped[str] = mapped_column(String(64), primary_key=True) document_id: Mapped[str] = mapped_column( String(36), ForeignKey("documents.id", ondelete="CASCADE"), nullable=False, index=True ) chunk_index: Mapped[int] = mapped_column(Integer, nullable=False) content: Mapped[str] = mapped_column(Text, nullable=False) embedding = mapped_column(Vector(get_settings().embed_dim), nullable=False) # 768 tsv = mapped_column(TSVECTOR, Computed("to_tsvector('english', content)", persisted=True)) __table_args__ = ( Index("ix_chunks_tsv", "tsv", postgresql_using="gin"), Index("ix_chunks_embedding", "embedding", postgresql_using="hnsw", postgresql_ops={"embedding": "vector_cosine_ops"}), ) ``` Deleting a document is now one statement. ON DELETE CASCADE removes its chunks, which used to need a separate call to Chroma. My first keyword search used websearch_to_tsquery, the function most Postgres full-text examples reach for. It found nothing for most questions. The psql output shows why: Two things go wrong. First, websearch_to_tsquery ANDs every term, so a full natural-language question only matches a chunk that contains *all* of its words. That's fine for a search box, but questions aren't search-box queries. Second, Postgres's parser treats ERR-4021 as the word err followed by the number -4021, minus sign included. A query I built in Python by splitting on punctuation sent 4021, which never matches -4021. The fix was to stop tokenizing in Python at all. The question goes through the same to_tsvector('english', ...) call as the documents, and the resulting lexemes are joined with OR. ts_rank_cd then ranks chunks that share more of them higher, which behaves a lot like BM25: ``` python def or_tsquery(question: str): lexemes = func.unnest(func.to_tsvector("english", question)).table_valued( "lexeme", "positions", "weights" ) return ( select(func.to_tsquery( "simple", func.coalesce(func.string_agg(func.quote_literal(column("lexeme")), " | "), ""), )) .select_from(lexemes) .scalar_subquery() ) ``` Now the ERR-4021 chunk scores 0.2 and ERR-4012 scores 0.1, because only one of them contains -4021. Product codes like NV-220X work the same way. A question made only of stopwords produces an empty query and matches nothing, which is the behavior you want. Vector search returns cosine similarities between 0 and 1. Keyword search returns ts_rank_cd scores, which are on a completely different scale. Normalizing and adding them is fiddly and fragile. Reciprocal Rank Fusion avoids the problem by ignoring scores entirely and using only rank. Each chunk gets 1 / (k + rank) from every list it appears in, with k = 60: ``` php def reciprocal_rank_fusion(result_lists: list[list[Candidate]], k: int = 60) -> list[Candidate]: fused: dict[str, Candidate] = {} for results in result_lists: for rank, candidate in enumerate(results, start=1): merged = fused.get(candidate.chunk_id) if merged is None: merged = Candidate( chunk_id=candidate.chunk_id, document_id=candidate.document_id, filename=candidate.filename, chunk_index=candidate.chunk_index, content=candidate.content, scores={"rrf": 0.0}, ) fused[candidate.chunk_id] = merged merged.scores.update(candidate.scores) merged.scores["rrf"] += 1.0 / (k + rank) return sorted(fused.values(), key=lambda c: c.scores["rrf"], reverse=True) ``` A chunk that ranks well in both lists beats one that’s first in only one. Both searches run concurrently with asyncio.gather, each in its own database session, and each returns 20 candidates for the fusion step. Both searches score the question and a chunk separately. A cross-encoder reads them *together* and outputs a single relevance score, which is much more accurate but too slow to run over a whole corpus. So it only runs over the fused candidates. I used ms-marco-MiniLM-L-6-v2. It's small (about 90 MB), runs on a CPU in milliseconds per pair, and costs nothing. Two details mattered in an async API: The model is an optional install (pip install -e ".[rerank]"), because it pulls in PyTorch. Without it you can still run RETRIEVAL_MODE=hybrid. The original golden set had 8 questions over 3 documents, and every mode already scored 100% on it. It couldn’t tell the modes apart. So I added 5 documents designed to be hard, and grew the set to 22 questions: I also changed what counts as a hit. “The right file was retrieved” is too generous when the 2025 and 2026 policies sit in different files but the answer depends on picking the right one. Each case now has an evidence string, and a hit means the retrieved *chunk* contains it. The runner reports hit@1, hit@4, and MRR (mean reciprocal rank) per mode: Hybrid search alone lifts hit@1 from 73% to 86%, and the reranker takes it to 91%. MRR goes from 0.864 to 0.936. The gains are exactly where I expected them: It’s not all wins. Two cases got worse with the reranker: damage-report fell from #1 to #3, and travel-2026-per-diem from #2 to #4. MiniLM is a small model trained on web search queries, and it has blind spots. A bigger reranker such as bge-reranker-base is one setting away, and the eval will show whether it's worth the extra latency. For answers, each case is also scored on whether the response contains the expected fact and whether an LLM judge finds it faithful to the retrieved context. With hybrid + rerank retrieval, all 22 passed both checks. The free tier was the most practical constraint of the project. Gemini’s free tier allows 5 chat requests per minute and only **20 per day** on gemini-3.6-flash. One answer-eval run needs 44 requests: an answer and a judge call per case. My first run died at case 8 on the per-minute limit, and the second ran out of the daily quota. Three changes made it workable: Retrieval evals are effectively free: 22 questions need 22 embedding calls and no chat calls. That’s the part I can run on every change. I didn’t want to mock the database. The interesting bugs here (tokenization, index types, cascade deletes) only show up in real Postgres. The test suite creates a separate rag_test database, builds the schema, and inserts chunks with *synthetic* embeddings: hand-made vectors that point along chosen axes, so a test can control exactly which chunk is "closest" without calling Gemini. The key test builds the exact failure from the intro. The query vector points at ERR-4012, but the question names ERR-4021. Vector search alone misses it, hybrid search brings it into the top results, and the reranker puts it first. One test failure taught me something about RRF. My first fixture had only two runbook chunks. ERR-4012 was #1 by vector and #2 by keyword, and ERR-4021 was the reverse, so both got *exactly* the same RRF score and the winner came down to sort order. That isn’t a bug in RRF. It’s what RRF should do when the two signals disagree evenly, and it’s the reason for having a reranker at the end. I rewrote the fixture to reflect the real scenario, where vector search ranks the right chunk low rather than a close second. I built this with an AI coding assistant (Claude Code) doing much of the implementation, while I made the decisions: Postgres instead of a dedicated vector database, a local reranker instead of a paid API, chunk-level evidence instead of file-level hits, and a lite model for evals rather than paying for quota. The assistant wrote code and tests, ran them, and reported failures back. The tokenization bug, the asyncpg codec conflict, and the RRF tie all came out of that loop of running real tests and reading real output. The corpus is still small: 8 documents and 17 chunks. Every mode finds the evidence somewhere in its top 4, so hit@4 is 100% across the board, and only hit@1 and MRR separate them. The React UI didn’t change in this round. And 22 out of 22 on answers says more about how easy the questions are than about the model. Next up: The code is on GitHub: [github.com/Eric-Souza/gemini-rag-doc-processing](https://github.com/Eric-Souza/gemini-rag-doc-processing). Run docker compose up -d postgres, add a free Gemini key, and python -m evals.run_evals --seed --retrieval-only will reproduce the comparison table above. If you’ve tuned hybrid search in production, I’d like to hear what worked for you, especially which reranker you ended up with. Leave a comment. [Close enough isn’t good enough: teaching an AI assistant to find the exact answer](https://blog.devgenius.io/close-enough-isnt-good-enough-teaching-an-ai-assistant-to-find-the-exact-answer-1d127caad4b7) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.