LLMOps for RAG Systems — Production Checklist

A developer's production checklist for LLMOps in RAG systems identifies weak retrieval, lack of reranking, poor prompt/version control, insufficient eval coverage, and blunt cost controls as common root causes of hallucinations and rising inference bills. The checklist recommends hybrid retrieval with reciprocal rank fusion, conditional cross-encoder reranking, prompt versioning, golden-set evaluation gates, and model routing with caching to ship safer, cheaper features in 2–4 sprints.

If your LLM feature starts hallucinating or your inference bill doubles overnight, the model is rarely the root cause. In production RAG systems the usual culprits are missing operational practices: weak retrieval, no reranker, no prompt/version control, poor eval coverage, and blunt cost controls. Treating RAG as “embed → nearest neighbors → prompt” is how teams discover outages, bad answers, and runaway spend. This article gives a compact, practical LLMOps for RAG systems checklist I use with engineering teams to ship safer, cheaper features in 2–4 sprints. Why: Dense embeddings find semantic matches; sparse BM25 preserves exact anchors IDs, dates, SKUs . Together they cover complementary failure modes and raise recall@k by ~15–30% in benchmarks. How: Run both retrievers in parallel, fetch top-K from each e.g., 20 , then fuse. Reciprocal Rank Fusion RRF is robust because it uses ranks not scores. Practical tip: Keep the same embedding model for query and docs and version it; store model id with each vector. Why: Bi-encoder retrieval is fast but approximate. A cross-encoder reads the query and candidate jointly and produces much better relevance ordering — the usual production pattern is 100 → 20 → 5: fetch 100 candidates with a bi-encoder, run a lightweight reranker on top 20, and pass the top 3–5 chunks to the generator. Cost control: Make reranking conditional. If the fused top score exceeds a confidence threshold, skip the cross-encoder. Impact: Teams report 50–60% reductions in obvious hallucinations after adding reranking plus stricter prompting. Why: Prompts change often. Store canonical prompt templates, fallbacks, per-model variable bindings, and tag versions v1, v2 . Tie deployments to a prompt hash so rollbacks are deterministic. How: Keep prompts in the repo or a small service prompt-registry , include CI checks that run the eval harness against the new prompt before promotion. Why: You need measurable gates. Build a golden set 200–500 representative Q&A pairs and automate daily smoke tests and weekly adversarial runs. Track precision@k, recall@k, faithfulness does each claim cite a retrieved chunk? , and safety flags. How: Run end-to-end CI checks on PRs that touch retrieval, chunking, prompts or model versions. Block deploys on regressions beyond a chosen threshold e.g. 3% drop in recall@5 . Why: Most inference cost is generation. Route cheap, high-recall queries to smaller models and reserve large models for long-context or high-confidence scoring. Cache reranker outputs and semantic-cache generated answers for repeated queries. How: Use an intent/complexity classifier or cheap heuristics query length, presence of numbers/IDs, metadata to choose a model. Cache reranker scores keyed by query hash, candidate set hash and only call cross-encoder when the cache misses or when confidence is low. Sprint 1 1–2 sprints Sprint 2 1 sprint Sprint 3 optional python simplified; replace with your DB/SDK calls from sentence transformers import SentenceTransformer, CrossEncoder embedder = SentenceTransformer 'all-mpnet-base-v2' cross = CrossEncoder 'cross-encoder/ms-marco-MiniLM-L-6-v2' def hybrid candidates query, vector db, bm25 index, top k=20 : q emb = embedder.encode query dense = vector db.search q emb, k=top k returns id, score , ... sparse = bm25 index.search query, k=top k returns id, score , ... merge by rank using RRF fused = reciprocal rank fusion dense, sparse, k const=60 return fused :top k def conditional rerank query, candidates, rerank threshold=0.35, rerank top n=5 : top score = candidates 0 .fused score if top score rerank threshold: return candidates :rerank top n build pairs and rerank pairs = query, c.text for c in candidates :100 rerank scores = cross.predict pairs for c, s in zip candidates :100 , rerank scores : c.score = s candidates.sort key=lambda c: c.score, reverse=True return candidates :rerank top n This pattern keeps the reranker budgeted and predictable while improving top-K precision. LLMOps for RAG systems isn’t an academic checklist — it’s engineering. Focus on deterministic pipelines, measurable gates RAGAS , and a staged rollout: hybrid retrieval, reranking, prompt/version control, automated evals, and then cost routing. Do that and the model will behave. Which one of the five would you tackle first on your codebase?