{"slug": "i-built-a-python-sdk-to-debug-rag-pipelines", "title": "I Built a Python SDK to Debug RAG Pipelines", "summary": "A developer released rag-debugger, an open-source MIT-licensed Python SDK that intercepts and inspects retrieval-augmented generation (RAG) pipelines to surface missing knowledge-base coverage. The tool, installable via \"pip install rag-debugger-amine,\" decomposes queries into atomic sub-intents, scores each independently, and routes borderline scores between 0.60 and 0.75 through a reranker LLM call; its GapDetector defaults to a 0.65 threshold. rag-debugger uses Google Gemini by default with the gemini-3.5-flash-lite LLM and gemini-embedding-001 embeddings, supports LangChain, LlamaIndex, and custom retrievers, and ships a local dashboard at http://localhost:7842 that auto-refreshes every 10 seconds.", "body_md": "**Intercept, inspect, and fix your RAG retrieval pipeline.**\n\nMost RAG bugs aren't in your code — they're in your retrieval. Wrong chunks get\nselected, knowledge gaps go undetected, and you find out when users complain.\n`rag-debugger` gives you visibility into exactly what your vector DB returned,\nwhy it won, and what's missing from your knowledge base.\n\n```\npip install rag-debugger-amine\n```\n\nFor the local dashboard:\n\n```\npip install rag-debugger-amine[dashboard]\npython\nimport rag_debugger as rd\n\nrd.init(project=\"my-rag-app\")\nretriever = rd.wrap_retriever(your_retriever)\n```\n\nFind what your knowledge base is missing before your users do:\n\n``` python\nfrom rag_debugger import GeminiClient, GapDetector\n\nclient = GeminiClient()  # set GEMINI_API_KEY env var\ndetector = GapDetector(client)  # default threshold is 0.65\n\nchunks = your_retriever.get_relevant_documents(query)\nreport = detector.analyze(query, [{\"content\": c.page_content} for c in chunks])\n\nprint(report)\n# [GAP DETECTED] coverage=50%  worst_score=0.64  priority=0.50\n#   Missing: refund policy, iOS-specific cancellation\n#   Fix: Add docs covering refund eligibility and iOS cancellation flow.\n#     ✓ [0.71] how to cancel\n#     ✗ [0.64] how to get a refund\n```\n\nA query like *\"cancel my iOS subscription and get a refund\"* is really four\nquestions. Standard RAG scores the whole query — if cancellation chunks score\nhigh, the query looks covered. `rag-debugger` decomposes it into atomic\nsub-intents and scores each one independently, so a missing refund policy\nis always caught even when the cancellation docs are excellent.\n\nBorderline scores (0.60–0.75) are passed through a reranker — a lightweight LLM call that asks \"does this chunk actually answer this question?\" — so semantically similar but irrelevant chunks don't pass as covered.\n\nGroup multi-turn conversations under a single session to get a summary of retrieval quality across the whole interaction:\n\n```\nwith rd.session(id=\"conv-123\", user=\"user-42\") as s:\n    retriever.get_relevant_documents(\"first query\")\n    retriever.get_relevant_documents(\"follow-up query\")\n\nsummary = s.summary()\nprint(summary)\n# Session conv-123\n#   duration:    430ms\n#   events:      2\n#   avg score:   0.741\n#   worst score: 0.677\n#   gaps:        0 / 2\n```\n\nSessions are thread-safe — concurrent requests in a web app won't bleed into each other.\n\nVisualize retrieval events, chunk scores, and gap flags in a local web UI:\n\n```\nrd.dashboard()  # opens http://localhost:7842\n```\n\nThe dashboard shows:\n\n- Per-session summary — avg score, worst score, gap count\n- Per-event chunk score bars with content preview\n- Gap flags with missing topics and fix suggestions\n- Auto-refreshes every 10 seconds\n\nWorks with LangChain, LlamaIndex, and any custom pipeline:\n\n```\n# LangChain\nretriever = rd.wrap_retriever(vectorstore.as_retriever(), label=\"docs\")\n\n# LlamaIndex\nretriever = rd.wrap_retriever(index.as_retriever())\n\n# Custom object\nretriever = rd.wrap_retriever(my_retriever, method=\"fetch_docs\")\n```\n\n`rag-debugger` uses Google Gemini by default (free tier via\n[Google AI Studio](https://aistudio.google.com)):\n\n``` python\nfrom rag_debugger import GeminiClient\nclient = GeminiClient(api_key=\"...\")  # or set GEMINI_API_KEY env var\n```\n\nModels used:\n\n- LLM: `gemini-3.5-flash-lite`\n- Embeddings: `gemini-embedding-001`\n\nMIT", "url": "https://wpnews.pro/news/i-built-a-python-sdk-to-debug-rag-pipelines", "canonical_source": "https://github.com/mohamedaminefezzani/rag-debugger", "published_at": "2026-09-15 12:25:15+00:00", "updated_at": "2026-09-15 12:44:39.989500+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "ai-products", "mlops"], "entities": ["rag-debugger", "Google Gemini", "LangChain", "LlamaIndex", "gemini-3.5-flash-lite", "gemini-embedding-001", "Google AI Studio", "GapDetector"], "alternates": {"html": "https://wpnews.pro/news/i-built-a-python-sdk-to-debug-rag-pipelines", "markdown": "https://wpnews.pro/news/i-built-a-python-sdk-to-debug-rag-pipelines.md", "text": "https://wpnews.pro/news/i-built-a-python-sdk-to-debug-rag-pipelines.txt", "jsonld": "https://wpnews.pro/news/i-built-a-python-sdk-to-debug-rag-pipelines.jsonld"}}