How to Detect AI Hallucinations: Build an LLM Answer Checker (Full Code)

Here's an uncomfortable fact about AI answers: the lies are often the best-written part. I recently scored an AI-generated customer-support answer with my own evaluation tool — it earned 9/10 for clarity and 4/10 for faithfulness. Beautifully phrased, confidently wrong: it invented a 30-day refund window (the policy says 14), doubled a discount, and promised customers a free grinder that doesn't exist.

Fluent is not the same as true. If you're putting LLMs in front of customers — chatbots, support macros, document Q&A — you need a way to measure whether the output is grounded in your actual content. This guide builds one: an LLM Answer Checker that takes a source text and an AI answer and returns a trust report — faithfulness scores, a list of unsupported claims, and a suggested fix.

Try the live demo — it opens pre-loaded with the exact lying answer from the story above. One click and you'll watch all three fabrications get caught.

You can't regex your way to detecting hallucinations. The practical approach — used in serious eval frameworks — is LLM-as-judge: a second model call whose only job is auditing, with a rubric, against the source.

The key insight is in the framing. You don't ask "is this answer good?" You ask: "evaluate this answer against this source only — not against your own world knowledge." That one constraint turns a vague vibe-check into a grounded audit.

You are a rigorous AI-output auditor. You receive a SOURCE text and an
ANSWER that claims to be based on it. Evaluate the answer against the
source only — not against your own world knowledge.

Score each dimension as an integer 0-10:
- faithfulness: does every claim in the answer appear in or follow directly
  from the source? (10 = fully grounded; heavily penalize invented facts,
  numbers, names, or policies)
- relevance: does the answer address what the source covers?
- completeness: does it use the important relevant information available?
- clarity: is it well-organized and easy to understand?

Also produce:
- unsupported_claims: a list of strings, each quoting or closely
  paraphrasing a specific claim in the answer that the source does NOT
  support (empty list if none)
- verdict: one sentence, starting with "Trustworthy:", "Mostly reliable:",
  or "Unreliable:"
- suggested_fix: 1-2 sentences on how to correct the answer

Return JSON exactly:
{"faithfulness": n, "relevance": n, "completeness": n, "clarity": n,
 "unsupported_claims": [...], "verdict": "...", "suggested_fix": "..."}

Three design choices worth stealing:

The serving code is a small FastAPI app — the same free stack as my chatbot tutorial (Groq free tier + Vercel hosting). The essentials:

@app.post("/api/evaluate")
def evaluate(req: EvalRequest, request: Request) -> dict:
    context, answer = req.context.strip(), req.answer.strip()
    if not context or not answer:
        raise HTTPException(400, "Both the source text and the answer are required.")

    messages = [
        {"role": "system", "content": JUDGE_SYSTEM},
        {"role": "user", "content": f"SOURCE:\n{context}\n\nANSWER TO EVALUATE:\n{answer}"},
    ]

    def _call(strict_json: bool):
        kwargs = {"response_format": {"type": "json_object"}} if strict_json else {}
        return client.chat.completions.create(
            model=MODEL, messages=messages, max_tokens=3000,
            temperature=0.1, **kwargs,
        )

    try:
        try:
            resp = _call(strict_json=True)
        except Exception:
            resp = _call(strict_json=False)   # fallback: parse defensively
    except Exception as exc:
        raise HTTPException(502, f"LLM error: {exc}")

    return parse_report(resp.choices[0].message.content)

And one scoring decision in the report parser: the overall score weights faithfulness double —

overall = (faithfulness * 2 + relevance + completeness + clarity) / 5

— because a clear, relevant, complete answer that invents your refund policy is still a bad answer. Weight what you actually care about.

Trap 1 — reasoning models eat your token budget. My first deploy failed with a cryptic 400: Failed to generate JSON. The cause: modern "reasoning" models spend tokens thinking before they answer, and those tokens count against max_tokens. My 900-token budget was consumed by thinking; the JSON got truncated mid-object; strict JSON mode rejected it. Fix: a generous max_tokens=3000 for judge calls.

Trap 2 — strict JSON mode isn't guaranteed. Even with response_format={"type": "json_object"}, some model/provider combinations occasionally fail. The code above retries once without strict mode and parses defensively — stripping code fences, locating the outermost {…}.

Trap 3 — trust nothing in the parsed output. Clamp scores to 0–10 (max(0, min(10, int(x)))), tolerate missing keys, cap list lengths and string sizes. The judge is an LLM too; your parser is the adult in the room.

A checker is a diagnostic, not a cure. When faithfulness comes back low, the production fixes are:

If you want the grounded-chatbot side ready-made, that's exactly what my AI Chatbot Starter Kit ($29) is — a RAG chatbot that answers only from your documents, with the refusal behavior built in. The free tutorial code is the lighter version.

I'm Sammishra — an AI engineer and published researcher. I build chatbots, RAG pipelines, and AI evaluation tooling for businesses. See my work (including the live demos) at sammishra.hashnode.dev/work-with-me, or email sammishrakh.freelancer@gmail.com.