We cut RAG costs 5x without losing quality

Teams shipping RAG systems to production often see quality collapse and costs spiral, but semantic chunking, hybrid retrieval, and selective reranking can cut costs 5x while maintaining accuracy, according to a technical guide from an unnamed practitioner. The guide recommends replacing fixed-size chunking with semantic chunking using sentence embeddings and similarity thresholds to preserve document structure, and notes that top-10 retrieval results are garbage 40% of the time without proper tuning.

Ship RAG to production and watch it fail. Here's what works: semantic chunking, hybrid retrieval, reranking, and how to cut costs 5x. I've watched teams implement RAG systems, and here's what I see: they build something that works locally, ship it to production, and then it falls apart. The chatbot that could answer questions perfectly on a 10-document test set starts giving garbage responses on real data. Cost spirals because they're embedding everything multiple times. Latency blows out because they're reranking with a model that runs in 5 seconds per query. The problem isn't RAG itself. It's that most teams treat RAG as "throw documents in a vector DB and ask questions." That's not RAG at scale. That's a prototype. Real production RAG is about making a hundred decisions: How do you chunk documents without losing context? Which embedding model gives you accuracy without eating your budget? When do you rerank and when do you skip it? How do you handle the fact that your top-10 retrieval results are garbage 40% of the time? This post covers what I've learned from running RAG systems that actually work. Your chunking strategy determines everything downstream. Get it wrong and no amount of reranking or retrieval magic fixes it. Most teams use fixed-size chunks 512 tokens, overlap 50 or 100 . This is fine for learning. It's terrible for production. Here's why: documents have structure. A legal contract has sections. A technical doc has code snippets and explanations. A research paper has abstract, methodology, results. Fixed-size chunking ignores all of that. You end up with chunks that split sentences mid-thought, chunks that miss context because they're too small, or chunks that duplicate content excessively. 1. Fixed-Size Chunking Quick, Wrong python python from typing import List def fixed size chunking text: str, chunk size: int = 512, overlap: int = 50 - List str : """ Naive fixed-size chunking. Fast but loses document structure. """ chunks = step = chunk size - overlap for i in range 0, len text , step : chunk = text i : i + chunk size if len chunk 100: Skip tiny chunks chunks.append chunk return chunks This will split sentences, lose context, and waste tokens text = "The capital of France is Paris. It's known for the Eiffel Tower..." chunks = fixed size chunking text Result: 'The capital of France is Paris. It's known for the Eiffel ', 'Tower...' ^ Garbage. Sentence got split. 2. Semantic Chunking Better, What You Actually Need python python from sentence transformers import SentenceTransformer import numpy as np from typing import List, Tuple def semantic chunking text: str, model name: str = "all-MiniLM-L6-v2", similarity threshold: float = 0.5, min chunk size: int = 100, - List str : """ Split text at semantic boundaries. Algorithm: 1. Split text into sentences 2. Compute embeddings for each sentence 3. Calculate cosine similarity between adjacent sentences 4. Start new chunk when similarity drops below threshold This preserves meaning and document structure. """ model = SentenceTransformer model name Split into sentences in production, use nltk or spaCy sentences = text.split ". " sentences = s.strip + "." for s in sentences if s.strip if len sentences < 2: return text Get embeddings embeddings = model.encode sentences Calculate similarity between adjacent sentences chunks = current chunk = sentences 0 for i in range 1, len sentences : Cosine similarity similarity = np.dot embeddings i , embeddings i-1 / np.linalg.norm embeddings i np.linalg.norm embeddings i-1 If similarity drops topic changed , start new chunk if similarity < similarity threshold: chunk text = " ".join current chunk if len chunk text = min chunk size: chunks.append chunk text current chunk = sentences i else: current chunk.append sentences i Don't forget the last chunk if current chunk: chunk text = " ".join current chunk if len chunk text = min chunk size: chunks.append chunk text return chunks Usage text = """ The capital of France is Paris. It's known for the Eiffel Tower. The Eiffel Tower was built in 1889. It stands 330 meters tall. London is the capital of the UK. It has Big Ben and Westminster Abbey. """ chunks = semantic chunking text, similarity threshold=0.4 Result: 3 coherent chunks, no split sentences, preserves meaning In production, I use a hybrid approach: python python from typing import List, Dict, Any import re def production chunking text: str, source: str = "unknown", max chunk size: int = 1000, min chunk size: int = 100, - List Dict str, Any : """ Production RAG chunking. Strategy: 1. Preserve document structure sections, subsections 2. Chunk semantically within sections 3. Add metadata for filtering and ranking 4. Small overlap to catch cross-boundary information """ chunks = chunk id = 0 Split by markdown headers first preserve structure sections = re.split r'\n {1,3} ', text for section idx, section in enumerate sections : lines = section.split '\n' current chunk = current size = 0 for line idx, line in enumerate lines : line tokens = len line.split If adding this line exceeds max, save chunk and start new one if current size + line tokens max chunk size and current chunk: chunk text = '\n'.join current chunk if len chunk text min chunk size: chunks.append { "content": chunk text, "source": source, "section idx": section idx, "chunk id": chunk id, "length": len chunk text , "metadata": { "position": f"section {section idx} chunk {chunk id}", "type": "text" } } chunk id += 1 Keep last line for overlap current chunk = line current size = line tokens else: current chunk.append line current size += line tokens else: current chunk.append line current size += line tokens Save final chunk if current chunk: chunk text = '\n'.join current chunk if len chunk text min chunk size: chunks.append { "content": chunk text, "source": source, "section idx": section idx, "chunk id": chunk id, "length": len chunk text , "metadata": { "position": f"section {section idx} chunk {chunk id}", "type": "text" } } return chunks What I've learned: This is where most teams leak money. They pick an expensive embedding model and embed everything. Twice. On updates they re-embed everything. Embedding costs matter: For a corpus of 1 million documents at 500 tokens each: The local model is 1-2% worse at retrieval than the large model. That's acceptable when you have a reranker. The large model won't save you if your chunks are garbage. python python import os from typing import List, Dict import time class EmbeddingCostCalculator: """Track and predict embedding costs.""" COSTS = { "text-embedding-3-small": 0.02 / 1 000 000, per token "text-embedding-3-large": 0.08 / 1 000 000, "local": 0, free but compute cost } def init self, model: str = "text-embedding-3-small" : self.model = model self.total tokens = 0 self.total cost = 0 def calculate cost self, tokens: int - float: """Calculate cost for embedding N tokens.""" cost = tokens self.COSTS.get self.model, 0 self.total tokens += tokens self.total cost += cost return cost def predict corpus cost self, num documents: int, avg tokens per doc: int - Dict: """Predict cost to embed entire corpus.""" total tokens = num documents avg tokens per doc cost = total tokens self.COSTS.get self.model, 0 return { "model": self.model, "num documents": num documents, "total tokens": total tokens, "cost": cost, "cost per document": cost / num documents if num documents 0 else 0, } def compare models self, num documents: int, avg tokens per doc: int - Dict: """Compare cost across models.""" results = {} for model in self.COSTS.keys : self.model = model results model = self.predict corpus cost num documents, avg tokens per doc return results Example: 100k documents, 500 tokens each calculator = EmbeddingCostCalculator comparison = calculator.compare models 100 000, 500 for model, costs in comparison.items : print f"{model}: ${costs 'cost' :,.2f}" Output: text-embedding-3-small: $1,000.00 text-embedding-3-large: $4,000.00 local: $0.00 My strategy: This is where 73% of RAG failures happen. You retrieve garbage, the LLM can't fix it. python python from typing import List, Tuple, Dict from dataclasses import dataclass import numpy as np @dataclass class RetrievalResult: content: str score: float source: str rerank score: float = None class HybridRetriever: """ Hybrid retrieval: combine vector search + BM25 keyword search. Why? Vector search is great for semantic meaning but misses keywords. BM25 catches keywords. Together they cover more ground. """ def init self, vector store, bm25 index : self.vector store = vector store Your vector DB Pinecone, Weaviate, etc self.bm25 index = bm25 index BM25 for keyword search def retrieve self, query: str, top k: int = 5 - List RetrievalResult : """ 1. Retrieve top 50 with vector search broad, semantic 2. Retrieve top 50 with BM25 keyword matches 3. Merge results by reciprocal rank fusion 4. Return top K """ Vector search semantic vector results = self.vector store.search query, limit=50 vector scores = {r 'id' : 51 - i / 51 for i, r in enumerate vector results } BM25 search keywords bm25 results = self.bm25 index.search query, limit=50 bm25 scores = {r 'id' : 51 - i / 51 for i, r in enumerate bm25 results } Combine using reciprocal rank fusion combined = {} for result id in set list vector scores.keys + list bm25 scores.keys : v score = vector scores.get result id, 0 b score = bm25 scores.get result id, 0 Weight: 60% semantic, 40% keyword combined result id = 0.6 v score + 0.4 b score Sort and return top K sorted results = sorted combined.items , key=lambda x: x 1 , reverse=True :top k return RetrievalResult content=self. get content result id , score=score, source=self. get source result id for result id, score in sorted results def get content self, result id: str - str: """Get chunk content by ID.""" return self.vector store.get result id 'content' def get source self, result id: str - str: """Get source document.""" return self.vector store.get result id .get 'source', 'unknown' Real numbers from a project we did: Hybrid retrieval got us to 82% recall on the first pass. Vector-only was 64%. BM25-only was 71%. The combination caught edge cases both missed. Retrieval finds candidates. Reranking orders them correctly. This is where the magic happens. python python from sentence transformers import CrossEncoder from typing import List, Dict import numpy as np class CrossEncoderReranker: """ Cross-encoder reranking. What it does: - Takes query + each candidate chunk - Scores how well they match 0-1 - Returns sorted by actual relevance, not just similarity Why it works: - Considers query AND content together - Catches semantic mismatches that vector search misses - Puts garbage at the bottom Trade-off: - Slow: 50-200ms per query depending on model - But worth it: quality improvement is 10-30% """ def init self, model name: str = "cross-encoder/mmarco-MiniLMv2-L12-H384-v1" : self.model = CrossEncoder model name def rerank self, query: str, candidates: List Dict , top k: int = 3, - List Dict : """ Rerank candidates by relevance to query. Args: query: User query candidates: List of retrieved chunks top k: Return top K results Returns: Reranked candidates with scores """ Prepare pairs for cross-encoder pairs = query, candidate 'content' for candidate in candidates Score all pairs scores = self.model.predict pairs Sort by score ranked = sorted zip candidates, scores , key=lambda x: x 1 , reverse=True :top k Add rerank scores to candidates results = for candidate, score in ranked: candidate 'rerank score' = float score results.append candidate return results Usage reranker = CrossEncoderReranker retrieved = {"content": "Paris is the capital of France", "score": 0.85}, {"content": "The Eiffel Tower is in Paris", "score": 0.83}, {"content": "France produces wine", "score": 0.72}, query = "What is the capital of France?" reranked = reranker.rerank query, retrieved, top k=2 Results are perfectly ordered now ^ This is crucial when you have 10k documents When to rerank: Here's how I architect RAG systems that actually scale: python python from typing import List, Dict, Any from dataclasses import dataclass import time @dataclass class RAGConfig: """Production RAG configuration.""" Chunking chunk size: int = 800 tokens chunk overlap: int = 100 tokens min chunk size: int = 50 tokens Retrieval initial retrieval k: int = 20 Get top 20 candidates rerank k: int = 3 Rerank to top 3 retrieval timeout: float = 5.0 seconds Embedding embedding model: str = "all-MiniLM-L6-v2" Local, fast embedding batch size: int = 128 Reranking rerank model: str = "cross-encoder/mmarco-MiniLMv2-L12-H384-v1" enable reranking: bool = True Cost tracking track costs: bool = True class ProductionRAG: """Complete RAG pipeline for production.""" def init self, config: RAGConfig : self.config = config self.retriever = None Your vector DB + BM25 self.reranker = None CrossEncoder if enabled self.metrics = { "queries processed": 0, "total latency": 0, "retrieval latency": 0, "reranking latency": 0, "tokens used": 0, } def query self, question: str, context limit: int = 3000 - Dict str, Any : """ Process query through full RAG pipeline. Returns dict with: - answer: LLM response - context: Retrieved chunks used - metrics: Timing and cost info """ start time = time.time Step 1: Retrieve candidates retrieval start = time.time candidates = self. retrieve question, k=self.config.initial retrieval k retrieval latency = time.time - retrieval start Step 2: Rerank if enabled reranking latency = 0 if self.config.enable reranking and len candidates self.config.rerank k: rerank start = time.time candidates = self. rerank question, candidates reranking latency = time.time - rerank start Step 3: Build context respecting token limit context = self. build context candidates, limit=context limit Step 4: Generate answer with context answer = self. generate answer question, context Track metrics total latency = time.time - start time self.metrics "queries processed" += 1 self.metrics "total latency" += total latency self.metrics "retrieval latency" += retrieval latency self.metrics "reranking latency" += reranking latency return { "answer": answer, "context chunks": len context , "metrics": { "total ms": round total latency 1000, 2 , "retrieval ms": round retrieval latency 1000, 2 , "reranking ms": round reranking latency 1000, 2 , "context size": len context , }, "sources": c 'source' for c in candidates :self.config.rerank k } def retrieve self, query: str, k: int - List Dict : """Hybrid retrieval: vector + BM25.""" Implementation depends on your vector DB This is pseudocode return self.retriever.hybrid search query, limit=k def rerank self, query: str, candidates: List Dict - List Dict : """Rerank using cross-encoder.""" return self.reranker.rerank query, candidates, top k=self.config.rerank k def build context self, chunks: List Dict , limit: int = 3000 - List Dict : """Build context string, respecting token limit.""" context = token count = 0 for chunk in chunks: chunk tokens = len chunk 'content' .split if token count + chunk tokens limit: break context.append chunk token count += chunk tokens return context def generate answer self, question: str, context: List Dict - str: """Generate answer using LLM + context.""" Call your LLM here This is pseudocode context str = "\n\n".join c 'content' for c in context prompt = f"""Use the following context to answer the question. Context: {context str} Question: {question} Answer:""" response = llm.generate prompt return response return "Answer would go here" def get metrics self - Dict: """Get performance metrics.""" avg latency = self.metrics "total latency" / self.metrics "queries processed" if self.metrics "queries processed" 0 else 0 return { "queries processed": self.metrics "queries processed" , "avg latency ms": round avg latency 1000, 2 , "avg retrieval ms": round self.metrics "retrieval latency" / self.metrics "queries processed" 1000, 2 if self.metrics "queries processed" 0 else 0, "avg reranking ms": round self.metrics "reranking latency" / self.metrics "queries processed" 1000, 2 if self.metrics "queries processed" 0 else 0, } Here's what we actually spent on a 500k document RAG system: Initial approach wrong : Optimized approach: Quality didn't drop. We went from 78% recall to 81% recall with better chunking + hybrid retrieval + local reranking. python class CostOptimization: """Track and optimize RAG costs.""" MONTHLY COSTS = { "embeddings": { "text-embedding-3-large": 400, 500k docs "text-embedding-3-small": 100, "local": 0, }, "retrieval": { "pinecone pro": 84, "weaviate cloud": 150, "weaviate selfhosted": 120, }, "reranking": { "cohere rerank": 150, 50k queries "api calls": 200, "local": 0, }, } @staticmethod def compare strategies - Dict: """Compare cost of different strategies.""" strategies = { "expensive": { "embedding": "text-embedding-3-large", "retrieval": "pinecone pro", "reranking": "cohere rerank", "cost": 400 + 84 + 150, }, "optimized": { "embedding": "local", "retrieval": "weaviate selfhosted", "reranking": "local", "cost": 0 + 120 + 0, }, } return strategies @staticmethod def quality vs cost - str: """What you get for your money.""" return """ Expensive $684/month : - 78% recall, 95% precision - 1ms retrieval latency - ~2ms reranking latency - Fully managed Optimized $120/month : - 81% recall, 94% precision - 5ms retrieval latency local - ~80ms reranking latency - Self-hosted dev time cost Better quality, 5.7x cheaper. Tradeoff: operational complexity. """ Problem 1: "Retrieval keeps returning irrelevant results" Check in this order: Code to diagnose: python python def diagnose retrieval query: str, ground truth chunk id: str : """Find why retrieval is failing.""" Step 1: Vector search ranking vector results = vector search query, k=50 vector rank = next i for i, r in enumerate vector results if r 'id' == ground truth chunk id , None print f"Vector search rank: {vector rank}" Should be < 5 Step 2: BM25 ranking bm25 results = bm25 search query, k=50 bm25 rank = next i for i, r in enumerate bm25 results if r 'id' == ground truth chunk id , None print f"BM25 rank: {bm25 rank}" Should be < 10 Step 3: Hybrid ranking hybrid results = hybrid search query, k=20 hybrid rank = next i for i, r in enumerate hybrid results if r 'id' == ground truth chunk id , None print f"Hybrid rank: {hybrid rank}" Should be < 5 If still not in top-5, rerank might save it If not in top-20 at all, problem is in retrieval, not ranking Problem 2: "Latency is too high 500ms " Find the bottleneck: python python def find latency bottleneck query: str : """Profile where time is being spent.""" import time start = time.time retrieved = retrieval step query retrieval time = time.time - start start = time.time reranked = reranking step retrieved reranking time = time.time - start start = time.time answer = llm step reranked, query llm time = time.time - start total = retrieval time + reranking time + llm time print f"Retrieval: {retrieval time 1000:.0f}ms {retrieval time/total 100:.0f}% " print f"Reranking: {reranking time 1000:.0f}ms {reranking time/total 100:.0f}% " print f"LLM: {llm time 1000:.0f}ms {llm time/total 100:.0f}% " print f"Total: {total 1000:.0f}ms" Optimize the biggest component Problem 3: "Cost keeps growing" Track what's actually expensive: python python def track costs : """Monitor cost drivers.""" Per query costs embeddings cost = num queries avg tokens per query embedding cost per token reranking cost = num queries reranking cost per query llm cost = num queries context tokens llm cost per token In production, usually LLM cost dominates 60% , not retrieval 10% So reducing context size from 2000 to 1000 tokens saves more than optimizing reranking print f"Embedding cost: ${embeddings cost}" print f"Reranking cost: ${reranking cost}" print f"LLM cost: ${llm cost}" print f"Total: ${embeddings cost + reranking cost + llm cost}" Before you ship RAG to production: RAG isn't magic. It's straightforward once you understand the tradeoffs. Some teams get it right from day one. Others iterate for months. The difference isn't usually talent — it's priorities. What separates working RAG from frustrating RAG: The teams building RAG systems that work aren't using fancier models or more expensive APIs. They're being precise about what they measure, ruthless about what they optimize, and honest about what doesn't work for their problem. That's it. No secret sauce.