# LLMOps for RAG Systems — Production Checklist

> Source: <https://dev.to/nainikmehta/llmops-for-rag-systems-production-checklist-1agh>
> Published: 2026-08-27 13:01:58+00:00

If your LLM feature starts hallucinating or your inference bill doubles overnight, the model is rarely the root cause. In production RAG systems the usual culprits are missing operational practices: weak retrieval, no reranker, no prompt/version control, poor eval coverage, and blunt cost controls. Treating RAG as “embed → nearest neighbors → prompt” is how teams discover outages, bad answers, and runaway spend.

This article gives a compact, practical LLMOps for RAG systems checklist I use with engineering teams to ship safer, cheaper features in 2–4 sprints.

Why: Dense embeddings find semantic matches; sparse BM25 preserves exact anchors (IDs, dates, SKUs). Together they cover complementary failure modes and raise recall@k by ~15–30% in benchmarks.

How: Run both retrievers in parallel, fetch top-K from each (e.g., 20), then fuse. Reciprocal Rank Fusion (RRF) is robust because it uses ranks not scores.

Practical tip: Keep the same embedding model for query and docs and version it; store model id with each vector.

Why: Bi-encoder retrieval is fast but approximate. A cross-encoder reads the query and candidate jointly and produces much better relevance ordering — the usual production pattern is 100 → 20 → 5: fetch 100 candidates with a bi-encoder, run a lightweight reranker on top 20, and pass the top 3–5 chunks to the generator.

Cost control: Make reranking conditional. If the fused top score exceeds a confidence threshold, skip the cross-encoder.

Impact: Teams report 50–60% reductions in obvious hallucinations after adding reranking plus stricter prompting.

Why: Prompts change often. Store canonical prompt templates, fallbacks, per-model variable bindings, and tag versions (v1, v2). Tie deployments to a prompt hash so rollbacks are deterministic.

How: Keep prompts in the repo or a small service (prompt-registry), include CI checks that run the eval harness against the new prompt before promotion.

Why: You need measurable gates. Build a golden set (200–500 representative Q&A pairs) and automate daily smoke tests and weekly adversarial runs. Track precision@k, recall@k, faithfulness (does each claim cite a retrieved chunk?), and safety flags.

How: Run end-to-end CI checks on PRs that touch retrieval, chunking, prompts or model versions. Block deploys on regressions beyond a chosen threshold (e.g. >3% drop in recall@5).

Why: Most inference cost is generation. Route cheap, high-recall queries to smaller models and reserve large models for long-context or high-confidence scoring. Cache reranker outputs and semantic-cache generated answers for repeated queries.

How: Use an intent/complexity classifier or cheap heuristics (query length, presence of numbers/IDs, metadata) to choose a model. Cache reranker scores keyed by (query_hash, candidate_set_hash) and only call cross-encoder when the cache misses or when confidence is low.

Sprint 1 (1–2 sprints)

Sprint 2 (1 sprint)

Sprint 3 (optional)

``` python
# simplified; replace with your DB/SDK calls
from sentence_transformers import SentenceTransformer, CrossEncoder

embedder = SentenceTransformer('all-mpnet-base-v2')
cross = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def hybrid_candidates(query, vector_db, bm25_index, top_k=20):
    q_emb = embedder.encode(query)
    dense = vector_db.search(q_emb, k=top_k)      # returns [(id, score), ...]
    sparse = bm25_index.search(query, k=top_k)    # returns [(id, score), ...]
    # merge by rank using RRF
    fused = reciprocal_rank_fusion(dense, sparse, k_const=60)
    return fused[:top_k]

def conditional_rerank(query, candidates, rerank_threshold=0.35, rerank_top_n=5):
    top_score = candidates[0].fused_score
    if top_score > rerank_threshold:
        return candidates[:rerank_top_n]
    # build pairs and rerank
    pairs = [(query, c.text) for c in candidates[:100]]
    rerank_scores = cross.predict(pairs)
    for c, s in zip(candidates[:100], rerank_scores):
        c.score = s
    candidates.sort(key=lambda c: c.score, reverse=True)
    return candidates[:rerank_top_n]
```

This pattern keeps the reranker budgeted and predictable while improving top-K precision.

LLMOps for RAG systems isn’t an academic checklist — it’s engineering. Focus on deterministic pipelines, measurable gates (RAGAS), and a staged rollout: hybrid retrieval, reranking, prompt/version control, automated evals, and then cost routing. Do that and the model will behave.

Which one of the five would you tackle first on your codebase?