When building production Retrieval-Augmented Generation (RAG) pipelines, the primary failure mode isn't retrieval latency or vector index size—it's silent hallucination.
LLMs frequently generate fluent, highly plausible-sounding claims that are completely unsupported by the retrieved reference chunks. In multi-agent systems and mission-critical workflows, accepting an ungrounded draft can corrupt downstream decision-making.
To solve this in cyclic multi-agent graphs, I built and published rag-critic on PyPI—a lightweight, adversarial hallucination auditor and citation grounding engine architected specifically for LangGraph.
Standard RAG architectures evaluate answers at a holistic paragraph level using an LLM-as-a-judge. This approach has three major flaws:
📦 Project Links & Open-Source Repositories
PyPI Package: pypi.org/project/rag-critic
GitHub Repository: github.com/samjanjua6/rag-critic
Author Engineering Portfolio: https://shaheerulhassan.com
Install directly from PyPI:
bash
pip install rag-critic
from rag_critic import RAGCritic
critic = RAGCritic()
result = critic.evaluate(
draft_response="Python was released in 1991 by Guido van Rossum. It was originally named Monty.",
retrieved_chunks=[
"Python is an interpreted programming language released in 1991 by Guido van Rossum."
]
)
print(f"Status: {result.status}") # REVISE / APPROVED
print(f"Grounding Score: {result.grounding_score:.2f}") # e.g. 0.50
print(f"Hallucination Index: {result.hallucination_index:.2f}")
for claim in result.claims:
print(f"[{claim.status.value}] {claim.text}")
from langgraph.graph import StateGraph, END
from rag_critic import create_critic_node, critic_router
builder = StateGraph(MyRAGState)
builder.add_node("retrieve", retrieve_docs)
builder.add_node("generate", generate_draft)
builder.add_node("critic", create_critic_node())
builder.add_edge("retrieve", "generate")
builder.add_edge("generate", "critic")
builder.add_conditional_edges(
"critic",
critic_router,
{
"approved": END, # Output is grounded; return to user
"rewrite": "generate", # Minor gaps; instruct re-writer
"re_retrieve": "retrieve", # Heavy hallucination; expand search
}
)