{"slug": "show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set", "title": "Show HN: Python script to evaluate your RAG pipeline against a golden set", "summary": "Infrixo Systems released rag-evaluation-starter, an open-source Python script that scores a RAG pipeline against a golden question set on five metrics in a single run: Retrieval Recall@K, Answer Faithfulness (token overlap, no LLM judge), Answer Relevance (cosine similarity between question and answer embeddings), Exact Match/Token F1, and Latency plus Cost. The tool runs locally with no SaaS, signup or mandatory API key, installs via pip from the GitHub repo, and requires users to supply only a retriever function and a generator function; the optional relevance metric downloads a roughly 400 MB model on first run. The project's guidance recommends building a 30-50 question golden set drawn from the last 3 months of support tickets, since fewer than eight entries lets a single bad answer swing metrics by 12 percentage points.", "body_md": "Most RAG systems look fine in demos. This tells you where yours breaks — before your users do.\n\nFive metrics in a single run. No SaaS. No signup. No mandatory API key.\n\n**Retrieval Recall@K** — Did the right document chunk actually come back in the top results? This is the most common failure point, and the one teams check last.\n\n**Answer Faithfulness** — Is the answer grounded in what was retrieved, or is the model making things up? Measured by token overlap — no LLM judge required.\n\n**Answer Relevance** — Does the generated answer actually address the question? A system can be grounded and still off-topic. Measured by cosine similarity between question and answer embeddings.\n\n**Exact Match / Token F1** — How closely does the answer match the reference? Useful for factual questions where the answer is a specific value: a price, a date, a policy rule.\n\n**Latency + Cost** — How long does each query take? How much will the LLM judge cost? Catches performance regressions before they reach production.\n\n```\ngit clone https://github.com/infrixo-systems/rag-evaluation-starter.git\ncd rag-evaluation-starter\npip install -r requirements.txt\npython rag_eval.py --golden examples/golden_set.json --no-embeddings\n```\n\nYou'll see a colour-coded table in your terminal within seconds. No configuration needed.\n\nTo include the cosine-similarity relevance metric (requires ~400 MB model download on first run):\n\n```\npython rag_eval.py --golden examples/golden_set.json\n```\n\nSupply two functions. That's it.\n\n``` python\nfrom rag_eval import load_golden_set, evaluate, summarise, print_rich_table\n\n# 1. Your retriever — returns [{text, source_id}, ...]\ndef my_retriever(question: str) -> list[dict]:\n    results = my_vectorstore.similarity_search(question, k=3)\n    return [{\"text\": r.page_content, \"source_id\": r.metadata[\"source\"]} for r in results]\n\n# 2. Your generator — returns an answer string\ndef my_generator(question: str, context: list[str]) -> str:\n    ctx = \"\\n\\n\".join(context)\n    response = openai_client.chat.completions.create(\n        model=\"gpt-4o-mini\",\n        messages=[\n            {\"role\": \"system\", \"content\": f\"Answer using only this context:\\n{ctx}\"},\n            {\"role\": \"user\",   \"content\": question},\n        ],\n    )\n    return response.choices[0].message.content\n\n# 3. Run\ngolden  = load_golden_set(\"examples/golden_set.json\")\nresults = evaluate(golden, my_retriever, my_generator, k=3)\nsummary = summarise(results)\nprint_rich_table(results, summary)\n```\n\nThe `embedder_fn` parameter is optional — it defaults to `sentence-transformers/all-MiniLM-L6-v2`. Pass your own if you want consistency with your production embeddings.\n\nThis is the section most teams skip — and why most RAG evals are useless.\n\n**Start with your support tickets.** The questions users ask when things go wrong are exactly the queries your system needs to handle. Look at the last 3 months. Pull out 30–50 questions that were hard to answer or answered incorrectly.\n\n**Group by type.** A good golden set has a mix:\n\n- **Easy (factual)** — single-chunk lookup, unambiguous phrasing. Establishes a baseline.\n- **Medium (multi-chunk)** — the answer requires combining information from two sources. Tests whether retrieval and synthesis both work.\n- **Hard (reasoning)** — ambiguous question, answer requires judgment over context. Reveals where the model starts guessing.\n- **Adversarial** — the answer is*not* in the knowledge base. Tests whether the system says \"I don't know\" or fabricates an answer. Include at least one.\n- **Edge cases** — questions at the boundary of what your system knows. These are the first ones to break when you update your docs.\n\n**Aim for 30–50 entries** before you trust the scores. Eight is enough to get started; fewer than that and a single bad answer swings your metrics by 12 percentage points.\n\n**Keep expected answers short and specific.** \"The rate limit is 100 requests per minute\" is a better expected answer than a paragraph — Token F1 scoring rewards precision.\n\n**Update the golden set when your docs change.** A stale golden set gives you false confidence. Add new entries every time you add a major feature to your product.\n\nThe `examples/golden_set.json` file in this repo uses a fictional billing API as the domain. Use it as a template for your own entries.\n\n| Score | Signal | Action | \n|---|---|---|\n| 0.8 – 1.0 | Retrieval is working | Focus on generation quality | \n| 0.5 – 0.8 | Some queries miss the right chunk | Adjust chunk size or embedding model | \n| < 0.5 | Retrieval is broken for many queries | Fix retrieval before touching generation | \n\nA Recall@3 of 0.6 means 40% of questions didn't surface the right source. No prompt engineering will fix that — your generator is being asked to answer from wrong context.\n\nLow faithfulness + high recall → the model is hallucinating even when given the right context. Tighten the system prompt.\n\nLow relevance + high faithfulness → the model is accurately summarising the wrong thing. Your retrieval is returning off-topic chunks.\n\nThe thresholds at the top of `rag_eval.py` are starting points, not ground truth. For a customer-facing support bot you might set FAIL at 0.7 for faithfulness. For an internal search tool you might tolerate 0.4. Edit the `THRESHOLDS` dict to match your requirements.\n\nIf your scores are consistently low and you're not sure which lever to pull, this is the kind of diagnostic work we cover in a [Foundation Check](https://infrixo.com/start?utm_source=github&utm_medium=readme&utm_campaign=rag-eval).\n\n**Retrieval Recall is low:**\nChunk your documents smaller (try 256 tokens with 64-token overlap). Switch to a retrieval-focused embedding model like `BAAI/bge-large-en-v1.5`. Add a CrossEncoder reranker on top of your ANN results. Read more: [Why RAG Systems Fail in Production](https://infrixo.com/insights/why-rag-systems-fail?utm_source=github&utm_medium=readme&utm_campaign=rag-eval).\n\n**Faithfulness is low:**\nStrengthen the system prompt with explicit grounding instructions. Reduce `max_tokens` to limit padding. Consider filtering retrieved chunks to a minimum relevance threshold before passing them to the generator.\n\n**Relevance is low:**\nYour retriever is returning chunks that are topically related but not directly answering the question. Add a reranker or try hybrid search (keyword + dense vector).\n\n**Token F1 is low on easy questions:**\nCheck whether your expected answers are phrased differently from how your documents are written. Update `expected_answer` fields to match what a well-behaved system would actually produce.\n\nThe default metrics require no API key. For more nuanced faithfulness and relevance scoring, enable LLM-as-judge:\n\n```\n# Using OpenAI\nOPENAI_API_KEY=sk-... python rag_eval.py --golden examples/golden_set.json --llm-judge\n\n# Using Anthropic\nANTHROPIC_API_KEY=sk-ant-... python rag_eval.py --golden examples/golden_set.json --llm-judge --judge-model claude-haiku-4-5-20251001\n```\n\nThe judge scores faithfulness and relevance on a 1–5 scale with a one-sentence reason per score. Results are clearly labelled `[LLM-judged]` in the output. A cost estimate is printed at the end of each run.\n\n**LLM judge bias.** GPT-4o-mini and Claude Haiku have positional and length biases. Use judge scores as a signal, not a ground truth. The token-overlap metrics are noisier but unbiased.\n\n**Golden sets go stale.** When you update your documentation, your golden set needs updating too. A test that always passes is not testing anything useful.\n\n**This is diagnostic, not continuous eval.** This tool is designed for point-in-time evaluation against a fixed set. It will tell you whether your current system passes a threshold; it won't tell you when something breaks in production. For continuous monitoring, look at [Ragas](https://docs.ragas.io), [LangSmith](https://smith.langchain.com), or [TruLens](https://www.trulens.org).\n\n**Token F1 penalises valid paraphrasing.** A generated answer of \"authentication requires a Bearer token\" and an expected answer of \"use Bearer token authentication\" will score below 1.0 despite being semantically equivalent. This is a known limitation of string-matching metrics.\n\nThis tool intentionally covers the minimum viable eval for a team that has nothing. When you need more:\n\n- **[Ragas](https://docs.ragas.io)** — the most comprehensive open-source RAG metrics framework. Use this when you need LLM-based metrics at scale.\n- **[LangSmith](https://smith.langchain.com)** — evaluation, tracing, and dataset management integrated with LangChain.\n- **[TruLens](https://www.trulens.org)** — feedback functions and a dashboard for LLM apps.\n- **[DeepEval](https://docs.confident-ai.com)** — pytest-style unit tests for LLM outputs.\n\n```\npython rag_eval.py --golden PATH [options]\n\nRequired:\n  --golden PATH         Path to golden_set.json\n\nOptions:\n  --k INT               Top-K for retrieval recall (default: 3)\n  --output PATH         Output path prefix (default: results)\n  --csv                 Also export results as CSV\n  --llm-judge           Enable LLM-as-judge scoring\n  --judge-model MODEL   Model for LLM judge (default: gpt-4o-mini)\n  --no-embeddings       Skip cosine-similarity relevance metric\n  --no-json             Skip saving results.json\nsentence-transformers>=2.0\ntiktoken>=0.5\nrich>=13.0\nnumpy>=1.24\n```\n\nOptional (for LLM judge): `openai>=1.0` or `anthropic>=0.20`\n\nSee [CONTRIBUTING.md](https://github.com/infrixo-systems/rag-evaluation-starter/blob/main/CONTRIBUTING.md). New metrics and example golden sets are especially welcome.\n\nMIT — see [LICENSE](https://github.com/infrixo-systems/rag-evaluation-starter/blob/main/LICENSE).\n\nEvaluation tells you where your RAG is broken. Guardrails stop those breaks from reaching your users.\n\nOnce your scores are green, the next layer is a production safety pipeline:\n\n| Layer | What it catches | \n|---|---|\n| Faithfulness gate | LLM answers that distort or ignore retrieved context | \n| Input guardrails | Prompt injection via user queries and retrieved documents | \n| Output guardrails | PII leaks, toxic responses, off-topic drift | \n| Citation attribution | Unverifiable claims with no traceable source | \n| Uncertainty abstention | Low-confidence answers that should never reach users | \n\nWe built a complete implementation of this stack — claim-level LLM-as-judge, Presidio PII redaction, consistency-based abstention, and Prometheus monitoring — documented across a technical playbook and 7 executed Jupyter notebooks.\n\n[Get the Production RAG Guardrails Playbook →](https://infrixo.gumroad.com/l/rag-guardrails)\n\nThe Core Playbook is free. The full implementation kit (Labs A–G) is $39.\n\n*Built by [Infrixo Systems](https://infrixo.com) · [hello@infrixo.com](mailto:hello@infrixo.com)*", "url": "https://wpnews.pro/news/show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set", "canonical_source": "https://github.com/infrixo-systems/rag-evaluation-starter", "published_at": "2026-10-24 15:17:44+00:00", "updated_at": "2026-10-24 15:41:44.289310+00:00", "lang": "en", "topics": ["ai-tools", "developer-tools", "large-language-models", "artificial-intelligence"], "entities": ["Infrixo Systems", "rag-evaluation-starter", "GitHub", "sentence-transformers/all-MiniLM-L6-v2", "OpenAI", "gpt-4o-mini"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set", "markdown": "https://wpnews.pro/news/show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set.md", "text": "https://wpnews.pro/news/show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set.txt", "jsonld": "https://wpnews.pro/news/show-hn-python-script-to-evaluate-your-rag-pipeline-against-a-golden-set.jsonld"}}