{"slug": "evaluate-and-debug-rag-pipelines-with-ragas", "title": "Evaluate and Debug RAG Pipelines with Ragas", "summary": "Ragas 0.4.3 introduces an evaluation harness that scores RAG pipeline outputs on faithfulness, context precision, and answer relevancy, using LLM judges to identify whether failures originate from the retriever or the generator. The tutorial, authored by Rachel Goldstein, demonstrates setup with Python 3.10–3.13, OpenAI API key, and specific package versions, costing about a cent per full run with gpt-4o-mini and text-embedding-3-small.", "body_md": "# Evaluate and Debug RAG Pipelines with Ragas\n\nScore faithfulness, context precision, and answer relevancy to tell whether your retriever or your generator is failing.\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)\n\n## What you'll build\n\nA small evaluation harness that scores your RAG pipeline's outputs on three [Ragas](https://docs.ragas.io/) metrics (faithfulness, context precision, answer relevancy) and tells you whether a bad answer came from the retriever or the generator.\n\n## Prerequisites\n\n- Python 3.10–3.13. Verified on 3.13; Ragas declares support for 3.9+.\n- Ragas 0.4.3 and openai 1.109.1, the current stable releases this was tested against. The code below uses the\n`ragas.metrics.collections`\n\nAPI introduced in the 0.3/0.4 line. It will not run on ragas 0.2.x. - langchain-community pinned below 0.4 (see step 1; ragas 0.4.3 breaks with 0.4.x).\n- An\n[OpenAI](https://platform.openai.com/)API key with a small amount of credit. The judge calls in this tutorial use`gpt-4o-mini`\n\nand`text-embedding-3-small`\n\n; a full run costs about a cent. - Commands are for macOS/Linux. On Windows, activate the venv with\n`.venv\\Scripts\\activate`\n\ninstead.\n\nA note on how this works: these metrics are LLM-judged. Ragas sends your question, contexts, and answer to a judge model that extracts claims and checks them. That's why you need an API key even though you're evaluating outputs you already have.\n\n## 1. Set up the project\n\n```\nmkdir rag-eval && cd rag-eval\npython3 -m venv .venv && source .venv/bin/activate\npip install ragas==0.4.3 openai \"langchain-community<0.4\"\nexport OPENAI_API_KEY=\"sk-...\"\n```\n\nThe `langchain-community<0.4`\n\npin matters. Ragas 0.4.3 imports a module that langchain-community 0.4.x removed, and without the pin the install succeeds but every import fails (see Troubleshooting).\n\n## 2. Capture pipeline outputs as an eval dataset\n\nRagas evaluates records with four fields: `user_input`\n\n(the question), `retrieved_contexts`\n\n(the chunks your retriever returned), `response`\n\n(what your generator produced), and `reference`\n\n(a known-good answer, needed only for context precision). In a real app you'd log these from your pipeline; here we hand-craft three samples with known failure modes so you can see each metric react.\n\nSave as `samples.json`\n\n:\n\n```\n[\n  {\n    \"id\": \"good\",\n    \"user_input\": \"What port does PostgreSQL listen on by default?\",\n    \"retrieved_contexts\": [\n      \"PostgreSQL listens on TCP port 5432 by default. The port is set by the port parameter in postgresql.conf.\",\n      \"To change the listen address, edit the listen_addresses parameter in postgresql.conf and restart the server.\"\n    ],\n    \"response\": \"PostgreSQL listens on port 5432 by default. You can change it via the port parameter in postgresql.conf.\",\n    \"reference\": \"PostgreSQL's default port is 5432, configured by the port parameter in postgresql.conf.\"\n  },\n  {\n    \"id\": \"hallucinated-answer\",\n    \"user_input\": \"How do I enable WAL archiving in PostgreSQL?\",\n    \"retrieved_contexts\": [\n      \"To enable WAL archiving, set wal_level to replica or higher and set archive_mode to on in postgresql.conf.\",\n      \"The archive_command parameter defines the shell command used to copy a completed WAL segment to archive storage.\"\n    ],\n    \"response\": \"Set wal_level to replica and archive_mode to on in postgresql.conf. Note that archive_mode defaults to on since PostgreSQL 16, so most installations already archive WAL automatically.\",\n    \"reference\": \"Enable WAL archiving by setting wal_level to replica or higher, archive_mode to on, and configuring archive_command in postgresql.conf.\"\n  },\n  {\n    \"id\": \"bad-retrieval\",\n    \"user_input\": \"What is the maximum identifier length in PostgreSQL?\",\n    \"retrieved_contexts\": [\n      \"VACUUM reclaims storage occupied by dead tuples and is run automatically by the autovacuum daemon.\",\n      \"The pg_dump utility creates logical backups of a single PostgreSQL database in script or archive formats.\"\n    ],\n    \"response\": \"PostgreSQL identifiers are limited to 63 bytes by default, set by the NAMEDATALEN constant at compile time.\",\n    \"reference\": \"The maximum identifier length in PostgreSQL is 63 bytes, determined by NAMEDATALEN minus one.\"\n  }\n]\n```\n\nSample two retrieves the right chunks but the generator invents a claim (archive_mode does not default to on). Sample three answers correctly from the model's own knowledge while the retriever returned junk, a failure that stays invisible until the model gets a question it can't answer from memory.\n\n## 3. Write the scoring script\n\nSave as `eval_rag.py`\n\n:\n\n``` python\nimport asyncio\nimport json\n\nfrom openai import AsyncOpenAI\nfrom ragas.embeddings.base import embedding_factory\nfrom ragas.llms import llm_factory\nfrom ragas.metrics.collections import AnswerRelevancy, ContextPrecision, Faithfulness\n\nwith open(\"samples.json\") as f:\n    samples = json.load(f)\n\nasync def main():\n    client = AsyncOpenAI()  # reads OPENAI_API_KEY\n    llm = llm_factory(\"gpt-4o-mini\", client=client)\n    embeddings = embedding_factory(\n        \"openai\", model=\"text-embedding-3-small\", client=client\n    )\n\n    faithfulness = Faithfulness(llm=llm)\n    context_precision = ContextPrecision(llm=llm)\n    answer_relevancy = AnswerRelevancy(llm=llm, embeddings=embeddings)\n\n    print(f\"{'sample':<22}{'faithfulness':>14}{'ctx_precision':>15}{'relevancy':>11}\")\n    for s in samples:\n        faith, precision, relevancy = await asyncio.gather(\n            faithfulness.ascore(\n                user_input=s[\"user_input\"],\n                response=s[\"response\"],\n                retrieved_contexts=s[\"retrieved_contexts\"],\n            ),\n            context_precision.ascore(\n                user_input=s[\"user_input\"],\n                reference=s[\"reference\"],\n                retrieved_contexts=s[\"retrieved_contexts\"],\n            ),\n            answer_relevancy.ascore(\n                user_input=s[\"user_input\"],\n                response=s[\"response\"],\n            ),\n        )\n        print(\n            f\"{s['id']:<22}{faith.value:>14.2f}\"\n            f\"{precision.value:>15.2f}{relevancy.value:>11.2f}\"\n        )\n\nasyncio.run(main())\n```\n\nEach metric only sees the fields it needs, which is the point: faithfulness checks the response against the retrieved contexts, context precision checks the contexts against the reference, and answer relevancy ignores the contexts entirely. The three metrics for one sample run concurrently via `asyncio.gather`\n\nsince they're independent API calls.\n\n## 4. Read the scores like a debugger\n\nEach metric isolates one component, so a low score points at a specific fix:\n\n- Low faithfulness, high context precision: the generator is inventing claims the context doesn't support. Tighten the prompt (\"answer only from the provided context\") or use a stronger generation model.\n- Low context precision: the retriever is the problem. Look at chunk size, embedding model, top-k, or add a reranker. Faithfulness may still be high if the model faithfully summarizes the wrong chunks.\n- Low answer relevancy: the response dodges the question. Usually a prompt problem (over-long boilerplate answers) or the query needs rewriting before retrieval.\n\nHigh relevancy with low faithfulness and low precision is the dangerous quadrant: confident, on-topic answers built from nothing.\n\n## Verify it works\n\n```\npython eval_rag.py\n```\n\nExpected output (judge-based scores wobble a few points between runs; the pattern is what you're checking):\n\n```\nsample                  faithfulness  ctx_precision  relevancy\ngood                            1.00           1.00       0.97\nhallucinated-answer             0.67           1.00       0.93\nbad-retrieval                   0.00           0.00       0.91\n```\n\nThe run takes 20–60 seconds. Each sample's scores match its planted failure: the hallucinated answer loses a third of its faithfulness score (one unsupported claim out of three), and the bad-retrieval sample shows 0.00 precision while relevancy stays high because the answer itself reads fine.\n\n## Troubleshooting\n\n** ModuleNotFoundError: No module named 'langchain_community.chat_models.vertexai'** on any\n\n`ragas`\n\nimport. Ragas 0.4.3 depends on langchain-community without an upper bound, and 0.4.x removed this module. Fix: `pip install \"langchain-community<0.4\"`\n\n(resolves to 0.3.31).** OpenAIError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable**. The key isn't in the environment of the shell running the script. Re-run\n\n`export OPENAI_API_KEY=\"sk-...\"`\n\nin the same terminal, or pass `AsyncOpenAI(api_key=...)`\n\nexplicitly.** ValueError: Collections metrics only support modern embeddings. Found: LangchainEmbeddingsWrapper.** You called the deprecated no-argument\n\n`embedding_factory()`\n\nfrom an older tutorial. Metrics from `ragas.metrics.collections`\n\nrequire the modern interface: `embedding_factory(\"openai\", model=\"text-embedding-3-small\", client=client)`\n\n.** TypeError: AnswerRelevancy.__init__() missing 1 required positional argument: 'embeddings'**. Unlike the other two metrics,\n\n`AnswerRelevancy`\n\nembeds generated questions to compare against the original, so it needs both `llm=`\n\nand `embeddings=`\n\n.## Next steps\n\nAdd `ContextRecall`\n\nand `FactualCorrectness`\n\n(both in `ragas.metrics.collections`\n\n, both need `reference`\n\n) to catch retrievers that miss chunks rather than fetch wrong ones. Replace the hand-written samples with real traces logged from your pipeline, or generate a synthetic test set from your documents with `ragas.testset.TestsetGenerator`\n\n. Once scores are stable, wire `eval_rag.py`\n\ninto CI and fail the build when faithfulness drops below your baseline, so a prompt tweak can't silently reintroduce hallucinations.\n\n## Sources & further reading\n\n-\n[Evaluate a simple RAG system](https://docs.ragas.io/en/stable/getstarted/rag_eval/)— docs.ragas.io -\n[Faithfulness metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/)— docs.ragas.io -\n[Context Precision metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/context_precision/)— docs.ragas.io -\n[Answer Relevancy metric](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/answer_relevance/)— docs.ragas.io -\n[ragas 0.4.3](https://pypi.org/project/ragas/)— pypi.org\n\n[Rachel Goldstein](https://sourcefeed.dev/u/rachel_goldstein)· Dev Tools Editor\n\nRachel has been embedded in the developer tooling ecosystem for nearly eight years, covering everything from IDE wars and package-manager drama to the quiet rise of AI-assisted coding. She has a soft spot for open-source maintainers and an unhealthy number of terminal emulators installed on a single laptop.\n\n## Discussion 0\n\nNo comments yet\n\nBe the first to weigh in.", "url": "https://wpnews.pro/news/evaluate-and-debug-rag-pipelines-with-ragas", "canonical_source": "https://sourcefeed.dev/a/evaluate-and-debug-rag-pipelines-with-ragas", "published_at": "2026-09-04 11:42:31+00:00", "updated_at": "2026-09-04 11:52:37.475341+00:00", "lang": "en", "topics": ["artificial-intelligence", "machine-learning", "large-language-models", "ai-tools", "ai-research"], "entities": ["Ragas", "Rachel Goldstein", "OpenAI", "gpt-4o-mini", "text-embedding-3-small", "Python", "PostgreSQL"], "alternates": {"html": "https://wpnews.pro/news/evaluate-and-debug-rag-pipelines-with-ragas", "markdown": "https://wpnews.pro/news/evaluate-and-debug-rag-pipelines-with-ragas.md", "text": "https://wpnews.pro/news/evaluate-and-debug-rag-pipelines-with-ragas.txt", "jsonld": "https://wpnews.pro/news/evaluate-and-debug-rag-pipelines-with-ragas.jsonld"}}