If your LLM feature starts hallucinating or your inference bill doubles overnight, the model is rarely the root cause. In production RAG systems the usual culprits are missing operational practices: weak retrieval, no reranker, no prompt/version control, poor eval coverage, and blunt cost controls. Treating RAG as “embed → nearest neighbors → prompt” is how teams discover outages, bad answers, and runaway spend.
This article gives a compact, practical LLMOps for RAG systems checklist I use with engineering teams to ship safer, cheaper features in 2–4 sprints.
Why: Dense embeddings find semantic matches; sparse BM25 preserves exact anchors (IDs, dates, SKUs). Together they cover complementary failure modes and raise recall@k by ~15–30% in benchmarks.
How: Run both retrievers in parallel, fetch top-K from each (e.g., 20), then fuse. Reciprocal Rank Fusion (RRF) is robust because it uses ranks not scores.
Practical tip: Keep the same embedding model for query and docs and version it; store model id with each vector.
Why: Bi-encoder retrieval is fast but approximate. A cross-encoder reads the query and candidate jointly and produces much better relevance ordering — the usual production pattern is 100 → 20 → 5: fetch 100 candidates with a bi-encoder, run a lightweight reranker on top 20, and pass the top 3–5 chunks to the generator.
Cost control: Make reranking conditional. If the fused top score exceeds a confidence threshold, skip the cross-encoder.
Impact: Teams report 50–60% reductions in obvious hallucinations after adding reranking plus stricter prompting.
Why: Prompts change often. Store canonical prompt templates, fallbacks, per-model variable bindings, and tag versions (v1, v2). Tie deployments to a prompt hash so rollbacks are deterministic.
How: Keep prompts in the repo or a small service (prompt-registry), include CI checks that run the eval harness against the new prompt before promotion.
Why: You need measurable gates. Build a golden set (200–500 representative Q&A pairs) and automate daily smoke tests and weekly adversarial runs. Track precision@k, recall@k, faithfulness (does each claim cite a retrieved chunk?), and safety flags.
How: Run end-to-end CI checks on PRs that touch retrieval, chunking, prompts or model versions. Block deploys on regressions beyond a chosen threshold (e.g. >3% drop in recall@5).
Why: Most inference cost is generation. Route cheap, high-recall queries to smaller models and reserve large models for long-context or high-confidence scoring. Cache reranker outputs and semantic-cache generated answers for repeated queries.
How: Use an intent/complexity classifier or cheap heuristics (query length, presence of numbers/IDs, metadata) to choose a model. Cache reranker scores keyed by (query_hash, candidate_set_hash) and only call cross-encoder when the cache misses or when confidence is low.
Sprint 1 (1–2 sprints)
Sprint 2 (1 sprint)
Sprint 3 (optional)
from sentence_transformers import SentenceTransformer, CrossEncoder
embedder = SentenceTransformer('all-mpnet-base-v2')
cross = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')
def hybrid_candidates(query, vector_db, bm25_index, top_k=20):
q_emb = embedder.encode(query)
dense = vector_db.search(q_emb, k=top_k) # returns [(id, score), ...]
sparse = bm25_index.search(query, k=top_k) # returns [(id, score), ...]
fused = reciprocal_rank_fusion(dense, sparse, k_const=60)
return fused[:top_k]
def conditional_rerank(query, candidates, rerank_threshold=0.35, rerank_top_n=5):
top_score = candidates[0].fused_score
if top_score > rerank_threshold:
return candidates[:rerank_top_n]
pairs = [(query, c.text) for c in candidates[:100]]
rerank_scores = cross.predict(pairs)
for c, s in zip(candidates[:100], rerank_scores):
c.score = s
candidates.sort(key=lambda c: c.score, reverse=True)
return candidates[:rerank_top_n]
This pattern keeps the reranker budgeted and predictable while improving top-K precision.
LLMOps for RAG systems isn’t an academic checklist — it’s engineering. Focus on deterministic pipelines, measurable gates (RAGAS), and a staged rollout: hybrid retrieval, reranking, prompt/version control, automated evals, and then cost routing. Do that and the model will behave.
Which one of the five would you tackle first on your codebase?