Building rag-critic: Factual Grounding & Adversarial Hallucination Auditing in LangGraph

A developer published rag-critic, an open-source PyPI package that audits RAG outputs for hallucination at the claim level rather than judging whole paragraphs. The tool scores grounding and hallucination indices, flags each claim as supported or ungrounded, and integrates with LangGraph as a validation node that routes drafts to approval, rewrite, or re-retrieval based on thresholds.

When building production Retrieval-Augmented Generation RAG pipelines, the primary failure mode isn't retrieval latency or vector index size— it's silent hallucination . LLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making. To solve this in cyclic multi-agent graphs, I built and published rag-critic https://pypi.org/project/rag-critic/ on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for LangGraph . Standard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws: 📦 Project Links & Open-Source Repositories PyPI Package: pypi.org/project/rag-critic //pypi.org/project/rag-critic GitHub Repository: github.com/samjanjua6/rag-critic //github.com/samjanjua6/rag-critic Author Engineering Portfolio: https://shaheerulhassan.com //shaheerulhassan.com Install directly from PyPI: python bash pip install rag-critic from rag critic import RAGCritic critic = RAGCritic Evaluate a draft answer against retrieved source chunks result = critic.evaluate draft response="Python was released in 1991 by Guido van Rossum. It was originally named Monty.", retrieved chunks= "Python is an interpreted programming language released in 1991 by Guido van Rossum." print f"Status: {result.status}" REVISE / APPROVED print f"Grounding Score: {result.grounding score:.2f}" e.g. 0.50 print f"Hallucination Index: {result.hallucination index:.2f}" Inspect claim-level verification for claim in result.claims: print f" {claim.status.value} {claim.text}" SUPPORTED Python was released in 1991 by Guido van Rossum. UNGROUNDED It was originally named Monty. from langgraph.graph import StateGraph, END from rag critic import create critic node, critic router builder = StateGraph MyRAGState Add standard generator and retriever nodes builder.add node "retrieve", retrieve docs builder.add node "generate", generate draft Add rag-critic as an adversarial validation checkpoint builder.add node "critic", create critic node builder.add edge "retrieve", "generate" builder.add edge "generate", "critic" Dynamic cyclic routing based on hallucination thresholds builder.add conditional edges "critic", critic router, { "approved": END, Output is grounded; return to user "rewrite": "generate", Minor gaps; instruct re-writer "re retrieve": "retrieve", Heavy hallucination; expand search }