How do you usually experiment with RAG pipelines?
I’ve been working with RAG recently and I’m trying to understand how people approach experimentation when they want to improve the quality of a system.
For example, if you change the retriever, chunking strategy, embeddings, reranker, or LLM, do you usually try several configurations and compare them? I’m curious what people use for this in practice. Is it mostly custom Python scripts/notebooks, tools like MLflow or W&B, or something else?
Also, how do you usually decide that one configuration is actually better than another? Do you rely mostly on automated evaluation metrics, manual testing, or a combination of both?
Would be interested to hear how people who have built more serious RAG systems handle this.