How to Detect AI Hallucinations: Build an LLM Answer Checker (Full Code)

A developer built an open-source LLM Answer Checker that uses an LLM-as-judge approach to audit AI-generated answers against a source text, returning faithfulness, relevance, completeness and clarity scores plus a list of unsupported claims and a suggested fix. The tool runs on a FastAPI backend with the Groq free tier and Vercel hosting, and weights faithfulness double in its overall score. In a demo, it caught an AI support answer that invented a 30-day refund window, doubled a discount and promised a nonexistent free grinder.

Here's an uncomfortable fact about AI answers: the lies are often the best-written part. I recently scored an AI-generated customer-support answer with my own evaluation tool — it earned 9/10 for clarity and 4/10 for faithfulness . Beautifully phrased, confidently wrong: it invented a 30-day refund window the policy says 14 , doubled a discount, and promised customers a free grinder that doesn't exist. Fluent is not the same as true. If you're putting LLMs in front of customers — chatbots, support macros, document Q&A — you need a way to measure whether the output is grounded in your actual content. This guide builds one: an LLM Answer Checker that takes a source text and an AI answer and returns a trust report — faithfulness scores, a list of unsupported claims, and a suggested fix. Try the live demo https://llm-answer-checker.vercel.app — it opens pre-loaded with the exact lying answer from the story above. One click and you'll watch all three fabrications get caught. You can't regex your way to detecting hallucinations. The practical approach — used in serious eval frameworks — is LLM-as-judge : a second model call whose only job is auditing, with a rubric, against the source. The key insight is in the framing. You don't ask "is this answer good?" You ask: "evaluate this answer against this source only — not against your own world knowledge." That one constraint turns a vague vibe-check into a grounded audit. You are a rigorous AI-output auditor. You receive a SOURCE text and an ANSWER that claims to be based on it. Evaluate the answer against the source only — not against your own world knowledge. Score each dimension as an integer 0-10: - faithfulness: does every claim in the answer appear in or follow directly from the source? 10 = fully grounded; heavily penalize invented facts, numbers, names, or policies - relevance: does the answer address what the source covers? - completeness: does it use the important relevant information available? - clarity: is it well-organized and easy to understand? Also produce: - unsupported claims: a list of strings, each quoting or closely paraphrasing a specific claim in the answer that the source does NOT support empty list if none - verdict: one sentence, starting with "Trustworthy:", "Mostly reliable:", or "Unreliable:" - suggested fix: 1-2 sentences on how to correct the answer Return JSON exactly: {"faithfulness": n, "relevance": n, "completeness": n, "clarity": n, "unsupported claims": ... , "verdict": "...", "suggested fix": "..."} Three design choices worth stealing: The serving code is a small FastAPI app — the same free stack as my chatbot tutorial https://sammishra.hashnode.dev/build-a-free-ai-support-chatbot-with-fastapi-groq-and-vercel-complete-guide Groq free tier + Vercel hosting . The essentials: php @app.post "/api/evaluate" def evaluate req: EvalRequest, request: Request - dict: context, answer = req.context.strip , req.answer.strip if not context or not answer: raise HTTPException 400, "Both the source text and the answer are required." messages = {"role": "system", "content": JUDGE SYSTEM}, {"role": "user", "content": f"SOURCE:\n{context}\n\nANSWER TO EVALUATE:\n{answer}"}, def call strict json: bool : kwargs = {"response format": {"type": "json object"}} if strict json else {} return client.chat.completions.create model=MODEL, messages=messages, max tokens=3000, temperature=0.1, kwargs, try: try: resp = call strict json=True except Exception: resp = call strict json=False fallback: parse defensively except Exception as exc: raise HTTPException 502, f"LLM error: {exc}" return parse report resp.choices 0 .message.content And one scoring decision in the report parser: the overall score weights faithfulness double — overall = faithfulness 2 + relevance + completeness + clarity / 5 — because a clear, relevant, complete answer that invents your refund policy is still a bad answer. Weight what you actually care about. Trap 1 — reasoning models eat your token budget. My first deploy failed with a cryptic 400: Failed to generate JSON . The cause: modern "reasoning" models spend tokens thinking before they answer, and those tokens count against max tokens . My 900-token budget was consumed by thinking; the JSON got truncated mid-object; strict JSON mode rejected it. Fix: a generous max tokens=3000 for judge calls. Trap 2 — strict JSON mode isn't guaranteed. Even with response format={"type": "json object"} , some model/provider combinations occasionally fail. The code above retries once without strict mode and parses defensively — stripping code fences, locating the outermost {…} . Trap 3 — trust nothing in the parsed output. Clamp scores to 0–10 max 0, min 10, int x , tolerate missing keys, cap list lengths and string sizes. The judge is an LLM too; your parser is the adult in the room. A checker is a diagnostic, not a cure. When faithfulness comes back low, the production fixes are: If you want the grounded-chatbot side ready-made, that's exactly what my AI Chatbot Starter Kit $29 https://sammishrakh.gumroad.com/l/mijcup is — a RAG chatbot that answers only from your documents, with the refusal behavior built in. The free tutorial code https://sammishrakh.gumroad.com/l/tgguty is the lighter version. I'm Sammishra — an AI engineer and published researcher. I build chatbots, RAG pipelines, and AI evaluation tooling for businesses. See my work including the live demos at sammishra.hashnode.dev/work-with-me https://sammishra.hashnode.dev/work-with-me , or email sammishrakh.freelancer@gmail.com mailto:sammishrakh.freelancer@gmail.com .