{"slug": "how-do-you-usually-experiment-with-rag-pipelines", "title": "How do you usually experiment with RAG pipelines?", "summary": "A developer seeking advice on experimenting with retrieval-augmented generation (RAG) pipelines asked the community how they compare configurations of retrievers, chunking strategies, embeddings, rerankers, and LLMs, and how they decide which setup is better. The question highlights the common use of custom Python scripts or notebooks, tools like MLflow or Weights & Biases, and a mix of automated metrics and manual testing.", "body_md": "How do you usually experiment with RAG pipelines?\n\nI’ve been working with RAG recently and I’m trying to understand how people approach experimentation when they want to improve the quality of a system.\n\nFor example, if you change the retriever, chunking strategy, embeddings, reranker, or LLM, do you usually try several configurations and compare them?\n\nI’m curious what people use for this in practice. Is it mostly custom Python scripts/notebooks, tools like MLflow or W&B, or something else?\n\nAlso, how do you usually decide that one configuration is actually better than another? Do you rely mostly on automated evaluation metrics, manual testing, or a combination of both?\n\nWould be interested to hear how people who have built more serious RAG systems handle this.", "url": "https://wpnews.pro/news/how-do-you-usually-experiment-with-rag-pipelines", "canonical_source": "https://discuss.huggingface.co/t/how-do-you-usually-experiment-with-rag-pipelines/178530#post_1", "published_at": "2026-08-09 18:42:00+00:00", "updated_at": "2026-08-09 18:43:46.502872+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "ai-research"], "entities": ["MLflow", "Weights & Biases"], "alternates": {"html": "https://wpnews.pro/news/how-do-you-usually-experiment-with-rag-pipelines", "markdown": "https://wpnews.pro/news/how-do-you-usually-experiment-with-rag-pipelines.md", "text": "https://wpnews.pro/news/how-do-you-usually-experiment-with-rag-pipelines.txt", "jsonld": "https://wpnews.pro/news/how-do-you-usually-experiment-with-rag-pipelines.jsonld"}}