{"slug": "optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40", "title": "Optimizing RAG at Scale: Chunking, Retrieval, and the Bayesian Search That Cut Latency 40%", "summary": "A developer rebuilt a retrieval-augmented generation (RAG) pipeline from first principles, achieving 95% recall@10 and cutting latency by 40% through a combination of document-type-aware chunking strategies, hybrid retrieval with reciprocal rank fusion, cross-encoder reranking, and Bayesian search optimization. The team moved from a naive 'semantic search + hope' approach to a measured, tunable system that outperforms standard RAG implementations in production.", "body_md": "*How we moved from \"semantic search + hope\" to a measured, tunable retrieval pipeline with 95% recall@10*\n\nEveryone ships RAG the same way: chunk by 512 tokens, embed with `text-embedding-3-small`\n\n, top-k=5, stuff into context. It works for demos.\n\nThen you hit production:\n\nWe rebuilt our retrieval layer from first principles. Here's what actually moves metrics.\n\n``` python\n# rag/chunking.py\nfrom abc import ABC, abstractmethod\nfrom dataclasses import dataclass\n\n@dataclass\nclass Chunk:\n    text: str\n    metadata: dict\n    token_count: int\n    chunk_id: str\n\nclass ChunkingStrategy(ABC):\n    @abstractmethod\n    def chunk(self, document: str, metadata: dict) -> list[Chunk]: ...\n\nclass FixedTokenChunker(ChunkingStrategy):\n    \"\"\"Baseline. Good for homogeneous content.\"\"\"\n    def __init__(self, chunk_size=512, overlap=50):\n        self.chunk_size = chunk_size\n        self.overlap = overlap\n\nclass RecursiveChunker(ChunkingStrategy):\n    \"\"\"Respects structure: markdown headers, code blocks, paragraphs.\"\"\"\n    def __init__(self, separators=[\"\\n## \", \"\\n### \", \"\\n\\n\", \"\\n\", \" \"], chunk_size=512):\n        self.separators = separators\n        self.chunk_size = chunk_size\n\nclass SemanticChunker(ChunkingStrategy):\n    \"\"\"Uses embedding similarity to find natural boundaries.\"\"\"\n    def __init__(self, model=\"text-embedding-3-small\", threshold=0.7):\n        self.model = model\n        self.threshold = threshold\n\nclass AgenticChunker(ChunkingStrategy):\n    \"\"\"LLM decides boundaries. Expensive but highest quality for complex docs.\"\"\"\n    def __init__(self, model=\"gpt-4o-mini\"):\n        self.model = model\n```\n\n**Our production config by document type:**\n\n| Document Type | Strategy | Chunk Size | Overlap | Recall@10 |\n|---|---|---|---|---|\n| Legal contracts | Recursive (clause-aware) | 1024 | 100 | 94% |\n| API reference | Recursive (function-aware) | 768 | 50 | 96% |\n| Support tickets | Semantic + conversation turns | 512 | 75 | 91% |\n| Internal wiki | Agentic (LLM) | 1500 | 200 | 97% |\n\nPure vector search misses exact matches (error codes, function names). Pure BM25 misses semantic matches. Hybrid wins.\n\n``` python\n# rag/retrieval.py\nclass HybridRetriever:\n    def __init__(self, vector_store, bm25_index, reranker, weights=(0.4, 0.3, 0.3)):\n        self.vector = vector_store\n        self.bm25 = bm25_index\n        self.reranker = reranker\n        self.weights = weights  # vector, bm25, reranker\n\n    async def retrieve(self, query: str, k=20, final_k=5):\n        # Stage 1: Parallel retrieval\n        vector_results = await self.vector.search(query, k=k)\n        bm25_results = await self.bm25.search(query, k=k)\n\n        # Stage 2: Reciprocal Rank Fusion\n        fused = self._rrf(vector_results, bm25_results, k=50)\n\n        # Stage 3: Cross-encoder rerank (top 50 → top 5)\n        reranked = await self.reranker.rerank(query, fused[:50])\n\n        return reranked[:final_k]\n\n    def _rrf(self, *result_lists, k=60):\n        \"\"\"Reciprocal Rank Fusion — no score calibration needed.\"\"\"\n        scores = defaultdict(float)\n        for results in result_lists:\n            for rank, doc in enumerate(results):\n                scores[doc.id] += 1 / (k + rank + 1)\n        return sorted(scores.items(), key=lambda x: -x[1])\n```\n\n**Why cross-encoder rerank?** Bi-encoder (embedding) similarity ≈ 0.75 correlation with relevance. Cross-encoder ≈ 0.92. The 50→5 funnel costs 50ms but gains 15% recall.\n\nUsers ask badly. Transform first.\n\n``` python\n# rag/query_transform.py\nclass QueryTransformer:\n    def __init__(self, llm_model=\"gpt-4o-mini\"):\n        self.llm = instructor.from_openai(AsyncOpenAI())\n\n    async def expand(self, query: str, context: dict = None) -> list[str]:\n        \"\"\"Generate multiple search queries from one user question.\"\"\"\n\n        class QuerySet(BaseModel):\n            queries: list[str] = Field(min_length=3, max_length=5)\n            reasoning: str\n\n        result = await self.llm.chat.completions.create(\n            model=self.model,\n            response_model=QuerySet,\n            messages=[\n                {\"role\": \"system\", \"content\": \"\"\"\nGenerate diverse search queries that collectively cover the user's intent.\nInclude: exact phrasing, synonyms, broader/narrower, hypothetical answer.\n\"\"\"},\n                {\"role\": \"user\", \"content\": f\"Original: {query}\\nContext: {context}\"}\n            ],\n            temperature=0.3,\n        )\n        return result.queries\n\n    async def decompose(self, query: str) -> list[str]:\n        \"\"\"Break multi-hop questions into sub-questions.\"\"\"\n\n        class SubQuestions(BaseModel):\n            questions: list[str]\n            needs_synthesis: bool\n\n        return await self.llm.chat.completions.create(\n            model=self.model,\n            response_model=SubQuestions,\n            messages=[...],\n        )\n```\n\n**Query expansion results:**\n\n`chunk_size=512`\n\n, `top_k=5`\n\n, `similarity_threshold=0.7`\n\n— who chose these?\n\nWe treat retrieval as a black-box function `f(chunk_size, overlap, top_k, weights) → recall@10, latency`\n\nand optimize with Bayesian search.\n\n``` python\n# rag/optimization.py\nimport optuna\nfrom dataclasses import dataclass\n\n@dataclass\nclass RetrievalConfig:\n    chunk_size: int\n    overlap: int\n    top_k: int\n    vector_weight: float\n    bm25_weight: float\n    rerank_top_k: int\n\ndef objective(trial: optuna.Trial) -> tuple[float, float]:\n    config = RetrievalConfig(\n        chunk_size=trial.suggest_categorical(\"chunk_size\", [256, 512, 768, 1024, 1536]),\n        overlap=trial.suggest_int(\"overlap\", 0, 200, step=25),\n        top_k=trial.suggest_int(\"top_k\", 5, 50, step=5),\n        vector_weight=trial.suggest_float(\"vector_weight\", 0.1, 0.8),\n        bm25_weight=trial.suggest_float(\"bm25_weight\", 0.1, 0.8),\n        rerank_top_k=trial.suggest_int(\"rerank_top_k\", 10, 100, step=10),\n    )\n\n    # Evaluate on golden set (200 queries)\n    recall, latency = evaluate_config(config, golden_set)\n\n    # Multi-objective: maximize recall, minimize latency\n    return recall, latency / 1000  # seconds\n\nstudy = optuna.create_study(\n    directions=[\"maximize\", \"minimize\"],\n    sampler=optuna.samplers.TPESampler(multivariate=True),\n)\nstudy.optimize(objective, n_trials=100, timeout=3600)  # 1 hour\n\n# Pareto frontier gives you the tradeoff curve\npareto = [t for t in study.trials if t.state == TrialState.COMPLETE]\n```\n\n**Our Pareto frontier (legal docs, 200-query golden set):**\n\n| Config | Recall@10 | Latency (p95) | Use Case |\n|---|---|---|---|\n| Conservative | 91% | 180ms | High-throughput API |\nBalanced (prod) |\n95% |\n320ms |\nDefault |\n| Aggressive | 97% | 580ms | High-stakes legal/medical |\n\n``` python\n# rag/metrics.py\nfrom prometheus_client import Histogram, Counter, Gauge\n\nRETRIEVAL_LATENCY = Histogram(\"rag_retrieval_latency_seconds\", \"End-to-end retrieval time\")\nRECALL_AT_K = Gauge(\"rag_recall_at_k\", \"Recall@k on golden set\", [\"k\"])\nQUERY_EXPANSION_COUNT = Counter(\"rag_query_expansions_total\", \"Number of expanded queries\")\nRERANKER_LATENCY = Histogram(\"rag_reranker_latency_seconds\", \"Cross-encoder rerank time\")\n\nclass InstrumentedRetriever(HybridRetriever):\n    async def retrieve(self, query, k=20, final_k=5):\n        with RETRIEVAL_LATENCY.time():\n            expanded = await self.transformer.expand(query)\n            QUERY_EXPANSION_COUNT.inc(len(expanded))\n\n            results = await super().retrieve(expanded, k, final_k)\n\n            # Track recall on sampled golden queries (1% of traffic)\n            if random.random() < 0.01:\n                RECALL_AT_K.labels(k=10).set(self._eval_recall(query, results))\n\n            return results\n```\n\n| Metric | Baseline (naive) | Optimized | Improvement |\n|---|---|---|---|\n| Recall@10 | 78% | 95% | +17 pp |\n| Latency p95 | 850ms | 320ms | -62% |\n| Hallucination rate | 12% | 3% | -75% |\n| Cost/query | $0.008 | $0.005 | -38% |\n\n**Retrieval is infrastructure, not afterthought.**\n\nYour users don't care about your embedding model. They care that the answer is right. Automated evaluation is how you guarantee that at scale.\n\n*Code: github.com/yourname/rag-eval-framework |\nDiscussion: Hacker News |\nFollow: @yourname*", "url": "https://wpnews.pro/news/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40", "canonical_source": "https://dev.to/imus_d7584cbc8ee9b0336256/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-latency-40-132c", "published_at": "2026-07-20 15:02:13+00:00", "updated_at": "2026-07-20 15:38:01.576229+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-infrastructure", "developer-tools"], "entities": ["OpenAI", "GPT-4o-mini", "text-embedding-3-small"], "alternates": {"html": "https://wpnews.pro/news/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40", "markdown": "https://wpnews.pro/news/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40.md", "text": "https://wpnews.pro/news/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40.txt", "jsonld": "https://wpnews.pro/news/optimizing-rag-at-scale-chunking-retrieval-and-the-bayesian-search-that-cut-40.jsonld"}}