{"slug": "building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in", "title": "Building rag-critic: Factual Grounding & Adversarial Hallucination Auditing in LangGraph", "summary": "A developer published rag-critic, an open-source PyPI package that audits RAG outputs for hallucination at the claim level rather than judging whole paragraphs. The tool scores grounding and hallucination indices, flags each claim as supported or ungrounded, and integrates with LangGraph as a validation node that routes drafts to approval, rewrite, or re-retrieval based on thresholds.", "body_md": "When building production Retrieval-Augmented Generation (RAG) pipelines, the primary failure mode isn't retrieval latency or vector index size—**it's silent hallucination**. \n\nLLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making.\n\nTo solve this in cyclic multi-agent graphs, I built and published [**`rag-critic`**](https://pypi.org/project/rag-critic/) on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for **LangGraph**.\n\nStandard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws:\n\n📦 Project Links & Open-Source Repositories\n\nPyPI Package: [pypi.org/project/rag-critic](//pypi.org/project/rag-critic)\n\nGitHub Repository: [github.com/samjanjua6/rag-critic](//github.com/samjanjua6/rag-critic)\n\nAuthor Engineering Portfolio: [https://shaheerulhassan.com](//shaheerulhassan.com)\n\nInstall directly from PyPI:\n\n``` python\nbash\npip install rag-critic\n\nfrom rag_critic import RAGCritic\n\ncritic = RAGCritic()\n\n# Evaluate a draft answer against retrieved source chunks\nresult = critic.evaluate(\n    draft_response=\"Python was released in 1991 by Guido van Rossum. It was originally named Monty.\",\n    retrieved_chunks=[\n        \"Python is an interpreted programming language released in 1991 by Guido van Rossum.\"\n    ]\n)\n\nprint(f\"Status:              {result.status}\")              # REVISE / APPROVED\nprint(f\"Grounding Score:      {result.grounding_score:.2f}\") # e.g. 0.50\nprint(f\"Hallucination Index:  {result.hallucination_index:.2f}\")\n\n# Inspect claim-level verification\nfor claim in result.claims:\n    print(f\"[{claim.status.value}] {claim.text}\")\n    # [SUPPORTED] Python was released in 1991 by Guido van Rossum.\n    # [UNGROUNDED] It was originally named Monty.\n\nfrom langgraph.graph import StateGraph, END\nfrom rag_critic import create_critic_node, critic_router\n\nbuilder = StateGraph(MyRAGState)\n\n# Add standard generator and retriever nodes\nbuilder.add_node(\"retrieve\", retrieve_docs)\nbuilder.add_node(\"generate\", generate_draft)\n\n# Add rag-critic as an adversarial validation checkpoint\nbuilder.add_node(\"critic\", create_critic_node())\n\nbuilder.add_edge(\"retrieve\", \"generate\")\nbuilder.add_edge(\"generate\", \"critic\")\n\n# Dynamic cyclic routing based on hallucination thresholds\nbuilder.add_conditional_edges(\n    \"critic\",\n    critic_router,\n    {\n        \"approved\": END,              # Output is grounded; return to user\n        \"rewrite\": \"generate\",        # Minor gaps; instruct re-writer\n        \"re_retrieve\": \"retrieve\",    # Heavy hallucination; expand search\n    }\n)\n```\n\n", "url": "https://wpnews.pro/news/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in", "canonical_source": "https://dev.to/samjanjua6/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in-langgraph-24hf", "published_at": "2026-10-23 20:58:29+00:00", "updated_at": "2026-10-23 21:09:55.097798+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "developer-tools", "ai-safety"], "entities": ["rag-critic", "LangGraph", "PyPI", "GitHub", "Shaheerul Hassan"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in", "markdown": "https://wpnews.pro/news/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in.md", "text": "https://wpnews.pro/news/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in.txt", "jsonld": "https://wpnews.pro/news/building-rag-critic-factual-grounding-adversarial-hallucination-auditing-in.jsonld"}}