Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%

A developer rebuilt a retrieval-augmented generation (RAG) pipeline from first principles, achieving 95% recall@10 and cutting latency by 40% through a combination of document-type-aware chunking strategies, hybrid retrieval with reciprocal rank fusion, cross-encoder reranking, and Bayesian search optimization. The team moved from a naive 'semantic search + hope' approach to a measured, tunable system that outperforms standard RAG implementations in production.

How we moved from "semantic search + hope" to a measured, tunable retrieval pipeline with 95% recall@10 Everyone ships RAG the same way: chunk by 512 tokens, embed with text-embedding-3-small , top-k=5, stuff into context. It works for demos. Then you hit production: We rebuilt our retrieval layer from first principles. Here's what actually moves metrics. python rag/chunking.py from abc import ABC, abstractmethod from dataclasses import dataclass @dataclass class Chunk: text: str metadata: dict token count: int chunk id: str class ChunkingStrategy ABC : @abstractmethod def chunk self, document: str, metadata: dict - list Chunk : ... class FixedTokenChunker ChunkingStrategy : """Baseline. Good for homogeneous content.""" def init self, chunk size=512, overlap=50 : self.chunk size = chunk size self.overlap = overlap class RecursiveChunker ChunkingStrategy : """Respects structure: markdown headers, code blocks, paragraphs.""" def init self, separators= "\n ", "\n ", "\n\n", "\n", " " , chunk size=512 : self.separators = separators self.chunk size = chunk size class SemanticChunker ChunkingStrategy : """Uses embedding similarity to find natural boundaries.""" def init self, model="text-embedding-3-small", threshold=0.7 : self.model = model self.threshold = threshold class AgenticChunker ChunkingStrategy : """LLM decides boundaries. Expensive but highest quality for complex docs.""" def init self, model="gpt-4o-mini" : self.model = model Our production config by document type: | Document Type | Strategy | Chunk Size | Overlap | Recall@10 | |---|---|---|---|---| | Legal contracts | Recursive clause-aware | 1024 | 100 | 94% | | API reference | Recursive function-aware | 768 | 50 | 96% | | Support tickets | Semantic + conversation turns | 512 | 75 | 91% | | Internal wiki | Agentic LLM | 1500 | 200 | 97% | Pure vector search misses exact matches error codes, function names . Pure BM25 misses semantic matches. Hybrid wins. python rag/retrieval.py class HybridRetriever: def init self, vector store, bm25 index, reranker, weights= 0.4, 0.3, 0.3 : self.vector = vector store self.bm25 = bm25 index self.reranker = reranker self.weights = weights vector, bm25, reranker async def retrieve self, query: str, k=20, final k=5 : Stage 1: Parallel retrieval vector results = await self.vector.search query, k=k bm25 results = await self.bm25.search query, k=k Stage 2: Reciprocal Rank Fusion fused = self. rrf vector results, bm25 results, k=50 Stage 3: Cross-encoder rerank top 50 → top 5 reranked = await self.reranker.rerank query, fused :50 return reranked :final k def rrf self, result lists, k=60 : """Reciprocal Rank Fusion — no score calibration needed.""" scores = defaultdict float for results in result lists: for rank, doc in enumerate results : scores doc.id += 1 / k + rank + 1 return sorted scores.items , key=lambda x: -x 1 Why cross-encoder rerank? Bi-encoder embedding similarity ≈ 0.75 correlation with relevance. Cross-encoder ≈ 0.92. The 50→5 funnel costs 50ms but gains 15% recall. Users ask badly. Transform first. python rag/query transform.py class QueryTransformer: def init self, llm model="gpt-4o-mini" : self.llm = instructor.from openai AsyncOpenAI async def expand self, query: str, context: dict = None - list str : """Generate multiple search queries from one user question.""" class QuerySet BaseModel : queries: list str = Field min length=3, max length=5 reasoning: str result = await self.llm.chat.completions.create model=self.model, response model=QuerySet, messages= {"role": "system", "content": """ Generate diverse search queries that collectively cover the user's intent. Include: exact phrasing, synonyms, broader/narrower, hypothetical answer. """}, {"role": "user", "content": f"Original: {query}\nContext: {context}"} , temperature=0.3, return result.queries async def decompose self, query: str - list str : """Break multi-hop questions into sub-questions.""" class SubQuestions BaseModel : questions: list str needs synthesis: bool return await self.llm.chat.completions.create model=self.model, response model=SubQuestions, messages= ... , Query expansion results: chunk size=512 , top k=5 , similarity threshold=0.7 — who chose these? We treat retrieval as a black-box function f chunk size, overlap, top k, weights → recall@10, latency and optimize with Bayesian search. python rag/optimization.py import optuna from dataclasses import dataclass @dataclass class RetrievalConfig: chunk size: int overlap: int top k: int vector weight: float bm25 weight: float rerank top k: int def objective trial: optuna.Trial - tuple float, float : config = RetrievalConfig chunk size=trial.suggest categorical "chunk size", 256, 512, 768, 1024, 1536 , overlap=trial.suggest int "overlap", 0, 200, step=25 , top k=trial.suggest int "top k", 5, 50, step=5 , vector weight=trial.suggest float "vector weight", 0.1, 0.8 , bm25 weight=trial.suggest float "bm25 weight", 0.1, 0.8 , rerank top k=trial.suggest int "rerank top k", 10, 100, step=10 , Evaluate on golden set 200 queries recall, latency = evaluate config config, golden set Multi-objective: maximize recall, minimize latency return recall, latency / 1000 seconds study = optuna.create study directions= "maximize", "minimize" , sampler=optuna.samplers.TPESampler multivariate=True , study.optimize objective, n trials=100, timeout=3600 1 hour Pareto frontier gives you the tradeoff curve pareto = t for t in study.trials if t.state == TrialState.COMPLETE Our Pareto frontier legal docs, 200-query golden set : | Config | Recall@10 | Latency p95 | Use Case | |---|---|---|---| | Conservative | 91% | 180ms | High-throughput API | Balanced prod | 95% | 320ms | Default | | Aggressive | 97% | 580ms | High-stakes legal/medical | python rag/metrics.py from prometheus client import Histogram, Counter, Gauge RETRIEVAL LATENCY = Histogram "rag retrieval latency seconds", "End-to-end retrieval time" RECALL AT K = Gauge "rag recall at k", "Recall@k on golden set", "k" QUERY EXPANSION COUNT = Counter "rag query expansions total", "Number of expanded queries" RERANKER LATENCY = Histogram "rag reranker latency seconds", "Cross-encoder rerank time" class InstrumentedRetriever HybridRetriever : async def retrieve self, query, k=20, final k=5 : with RETRIEVAL LATENCY.time : expanded = await self.transformer.expand query QUERY EXPANSION COUNT.inc len expanded results = await super .retrieve expanded, k, final k Track recall on sampled golden queries 1% of traffic if random.random < 0.01: RECALL AT K.labels k=10 .set self. eval recall query, results return results | Metric | Baseline naive | Optimized | Improvement | |---|---|---|---| | Recall@10 | 78% | 95% | +17 pp | | Latency p95 | 850ms | 320ms | -62% | | Hallucination rate | 12% | 3% | -75% | | Cost/query | $0.008 | $0.005 | -38% | Retrieval is infrastructure, not afterthought. Your users don't care about your embedding model. They care that the answer is right. Automated evaluation is how you guarantee that at scale. Code: github.com/yourname/rag-eval-framework | Discussion: Hacker News | Follow: @yourname