AI Can Write Your RAG Pipeline. It Can’t Teach You to Build a Good One. An open-source tool called rag-lab lets developers build ingestion and retrieval pipelines on a browser node canvas and test them against real benchmarks, scoring retrieval and answer quality separately, according to its author, who built it after failing to find a tool that made controlled RAG experiments easy. rag-lab scores retrieval with recall@1, recall@5, recall@10, nDCG@10 and MRR@10 against gold documents, and uses Claude as an LLM judge to grade generated answers against gold answers when a dataset provides them; SciFact supplies gold documents but no gold answers, while EnterpriseRAG-Bench provides both for 500 questions across 10 categories. The author argues that AI coding assistants can generate a plausible RAG pipeline in minutes, but choosing chunk sizes, hybrid search and rerankers requires measurement on a team's own documents and questions. Ask any AI coding assistant for a RAG pipeline and you will have one in a few minutes. It will load your documents, split them into chunks, embed them, put them in a vector store, retrieve the top matches for a question and hand them to an LLM. It runs, it answers, and the demo looks great. A year or two ago that took days of reading docs and gluing libraries together. Today it is a prompt. The code is no longer the hard part of Retrieval-Augmented Generation. So what is then? Let’s see. The usual path is tutorials. You follow a notebook, swap in your own PDFs, maybe try a different vector database. You learn the vocabulary: chunking, embeddings, hybrid search, reranking, agents. But a tutorial ends where the real questions start. Should your chunks be 300 characters or 1,200? Is hybrid search worth it on your data? Does the reranker everyone recommends actually help? Is the new pipeline better, or did it just answer the three questions you tried? None of those has a universal answer. They depend on your documents and your questions. The only way to answer them is to measure. An AI assistant writes a pipeline that is plausible. It picks sensible defaults: a popular embedding model, a common chunk size, a well-known reranker. Plausible is not the same as good for your data, and the assistant has no way to know the difference without measuring either. What separates a good RAG engineer from someone with working code is judgement: You cannot get that judgement by reading generated code. You get it by running experiments and looking at the results. The way to learn RAG is the way scientists learn anything: run controlled experiments. That needs four things. AI is still very useful in this loop. It can write the script for your experiment in seconds. What it cannot do is decide what the experiment shows. That part is yours, and it is the skill worth learning. I could not find a tool that made this loop easy, so I built one. rag-lab https://github.com/kkfaisal/rag-lab is an open-source playground that runs on your own machine. You build ingestion and retrieval pipelines on a node canvas in the browser, run them against real benchmarks, and see exactly what each change did. Here is how a RAG experiment flows through it: Three design choices make it a learning tool rather than a framework: A benchmark is an answer key. For every question it lists the documents that hold the answer the gold documents and, in some datasets, the right answer itself the gold answer . rag-lab scores two things separately: did retrieval find the gold documents, and does the generated answer match the gold answer. Each question in a dataset is one line like this: {"question id": "q1", "question": "…", "expected doc ids": "doc-17", "doc-42" , "gold answer": "…", "category": "…"}Auto JSON SciFact, for example, has gold documents but no gold answers: the claim “0-dimensional biomaterials show inductive properties.” has one gold document, 31715818. EnterpriseRAG-Bench has both, for 500 questions in 10 categories. Without gold answers you still get every retrieval score, for free and with no API key. Retrieval returns chunks, best first. rag-lab turns them into a ranked list of documents: each document takes the rank of its best chunk, and later chunks of the same document are dropped. That list is compared with the gold documents: A worked example: a question has one gold document, and retrieval ranks it third. Then recall@1 = 0, recall@5 = 100, recall@10 = 100, nDCG@10 = 50 and MRR@10 = 33.3. The document was found, but not first, and nDCG and MRR say so where recall does not. When a dataset has gold answers, an LLM judge Claude reads the question, the gold answer and the generated answer, and returns two scores: EnterpriseRAG-Bench questions can be graded by that benchmark’s own published judge instead. Every metric is computed per question, then averaged over all questions to give the run’s score, and per question category. Two runs are compared question by question on the same questions, which is how rag-lab can list the questions a change fixed and broke, and test whether the difference is real. Four pages turn those scores into something you can learn from. The examples below are real runs on the EnterpriseRAG-Bench dev subset 20 questions and SciFact. 1. The results matrix. The latest score for every target against every retrieval pipeline, coloured by score, with any metric picked from the menu. It shows things you would never guess from code. The same vector search pipeline gets recall@10 67.4 on the local store, pgvector and Elasticsearch, but 61.1 on OpenSearch, because each database searches its index in its own approximate way. Tick two cells to compare them. 2. One run. Everything that produced the score: each node’s settings, which steps came from the cache and how long the others took, the metric for each question category, and the tokens and cost. In this run, with Claude as an agent, recall@10 was 83.8 overall but 0 in the high level and info not found categories, which is where to look next. The 20 questions cost $0.75. 3. One question. The question, the generated answer, the judge’s verdict and reasoning, and the gold answer with the facts the judge checks. On the right, the gold documents, each tagged with where it got to: its retrieval rank, whether it reached the prompt, whether the answer cited it. Below them, the reranked chunks in order: green for a chunk from a gold document, amber for one that is not. This is where you see why a question failed. 4. Comparing two runs. Pick two runs and a metric. rag-lab shows both means, the difference, a 95% confidence interval, how many questions each run won, and a plain verdict from a paired permutation test. Then it lists the questions that were fixed, broken or unchanged. In this example, Claude as an agent scored 85% correct against 65% for a single pass: 20 points better, 4 questions fixed, none broken. Yet the verdict is “no significant difference p = 0.13 ”, because 20 questions are too few to tell. That one screen teaches more about evaluating RAG than any tutorial I have read. A lab with every tool still leaves you asking what to try next, so rag-lab comes with ten guided lessons. Each one is a small experiment with a goal score that is checked automatically against your runs: vector search, hybrid search, reranking, writing your own retriever, chunking, GraphRAG, single-pass answers and agentic RAG. The goals come from real measurements. Here is what each technique scores on SciFact 5,000 scientific abstracts, 300 questions with default settings, as recall@10 / nDCG@10: Two results surprised me, and they are exactly the kind of thing you only learn by measuring: An AI assistant would happily have added both to your pipeline. You need uv https://docs.astral.sh/uv/ and Node 20+. No API key is needed for the first lessons. git clone https://github.com/kkfaisal/rag-lab.git && cd rag-labuv syncuv run rag-lab quickstart --serveAuto Bash The quickstart downloads SciFact 4 MB , builds a BM25 index, answers all 300 questions and prints your first scores, about 15 seconds after the download. Then open http://127.0.0.1:8000, http://127.0.0.1:8000, go to Lessons , and start changing things. Let AI write your pipelines. Use rag-lab to learn which ones are actually good. The code is MIT-licensed and on GitHub: github.com/kkfaisal/rag-lab https://github.com/kkfaisal/rag-lab . Issues, ideas and new lessons are welcome. AI Can Write Your RAG Pipeline. It Can’t Teach You to Build a Good One. https://pub.towardsai.net/ai-can-write-your-rag-pipeline-it-cant-teach-you-to-build-a-good-one-05b15422964c was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.