Learn how to evaluate LLM outputs in production using system prompts, scored rubrics, and runtime guardrails to prevent hallucinations and ensure quality.
TL;DR: To evaluate LLM outputs in production, combine system prompts that define evaluation criteria, scored rubrics using LLM-as-a-judge for dimensions like correctness and relevance, and runtime guardrails that filter or flag unsafe outputs. This approach scales better than human review, adapts via prompt changes, and catches failures that status codes miss, as seen in the Air Canada chatbot case.
A 200 status code only confirms the server processed the request—it says nothing about whether the generated text is factual, safe, or useful. The Air Canada chatbot that invented a non-existent bereavement discount returned perfectly valid HTTP responses, yet the hallucinated policy led to a tribunal ruling against the airline. Production evaluation must therefore separate operational health (latency, error rates) from output quality (correctness, relevance, harmlessness).
Consider a typical API call that succeeds operationally but fails qualitatively:
import requests
response = requests.post(
"https://api.example.com/v1/chat",
json={"model": "gpt-4o", "messages": [{"role": "user", "content": "What is Air Canada's bereavement policy?"}]},
headers={"Authorization": "Bearer $KEY"}
)
print(response.status_code) # 200
print(response.json()["choices"][0]["message"]["content"])
A 200 status code and a well-formed JSON body mask a completely fabricated policy. To catch this, you need a separate evaluation layer that scores the output against a rubric. LLM-as-a-judge is a common approach, using a second model to assess the primary output on dimensions like factual accuracy.
eval_prompt = """
You are an evaluator. Score the following response on factual accuracy from 1 (completely fabricated) to 5 (fully accurate).
Response: "{response}"
Score:
"""
score = llm_eval(eval_prompt.format(response=chatbot_output))
if int(score) < 4:
alert_ops_team(chatbot_output, score)
This evaluation layer runs alongside every user-facing response, flagging low-quality outputs even when the system returns 200. Without it, you are measuring uptime while your model quietly erodes trust.
A system prompt for evaluation must explicitly define the LLM's role as an impartial judge, specify the exact output schema (e.g., JSON with a score and reasoning), and embed a detailed scoring rubric to ensure consistent, measurable assessments across all runs. Without this, LLM-as-a-judge outputs drift, undermining reliability.
Start by framing the evaluator's identity and task boundaries. Then provide a structured rubric with clear, mutually exclusive levels. For example, a relevance rubric might define: 1 (completely off-topic), 2 (tangential), 3 (partially relevant), 4 (mostly relevant), 5 (perfectly on-point). The prompt must also mandate a strict output format to enable automated parsing. This approach reduces prompt sensitivity, a known failure mode where small wording changes cause large score variations.
Here's a minimal system prompt for an LLM-as-a-judge evaluating answer relevance:
You are an impartial evaluation agent. Your task is to score the relevance of a generated answer to a given question.
Rubric:
1 - Completely irrelevant, does not address the question.
2 - Tangentially related but misses the core intent.
3 - Partially relevant, addresses some aspects but includes off-topic content.
4 - Mostly relevant, directly addresses the question with minor digressions.
5 - Perfectly relevant, concise and fully on-topic.
Output format: Return ONLY a valid JSON object with keys "score" (integer) and "reasoning" (string).
In code, you'd pass this system prompt alongside the user message containing the question and answer to evaluate. For example, using the OpenAI Python client:
import openai
response = openai.ChatCompletion.create(
model="gpt-4",
messages=[
{"role": "system", "content": system_prompt},
{"role": "user", "content": f"Question: {question}\nAnswer: {answer}"}
],
temperature=0.0 # deterministic scoring
)
Always set temperature to 0 for evaluation to maximize reproducibility. Version your system prompts in a prompt registry and run calibration tests against human-annotated samples to detect bias or inconsistency before production use.
LLM-as-a-Judge uses natural language rubrics to score outputs on dimensions like correctness, relevance, and tone; G-Eval chains evaluation steps to improve reliability. This approach replaces brittle string-matching with semantic assessment that scales across tasks and can be updated by simply changing the prompt.
A scored rubric defines criteria and a rating scale (e.g., 1–5) in plain language. The judge LLM receives the original query, the generated response, and the rubric, then returns a score with justification. For a customer support bot, a correctness rubric might read: "Score 5 if the answer is factually accurate and fully addresses the question; 1 if it contains hallucinated information." Relevance and tone rubrics follow the same pattern.
G-Eval extends this by first asking the LLM to generate detailed evaluation steps from the rubric, then using those steps to produce the final score. This chain-of-thought style reduces prompt sensitivity and yields more consistent ratings. The following example uses a simple Python function to call an LLM with a rubric, then a G-Eval style two-step chain:
import openai
def score_with_rubric(query, response, rubric):
prompt = f"""Evaluate the response based on the rubric.
Query: {query}
Response: {response}
Rubric: {rubric}
Provide a score (1-5) and a brief justification."""
result = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": prompt}],
temperature=0
)
return result.choices[0].message.content
python
def g_eval_score(query, response, rubric):
step_prompt = f"""Given the rubric, produce a numbered list of evaluation steps.
Rubric: {rubric}"""
steps = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": step_prompt}],
temperature=0
).choices[0].message.content
final_prompt = f"""Evaluate the response using the steps below.
Query: {query}
Response: {response}
Steps: {steps}
Provide a score (1-5) and justification."""
final = openai.chat.completions.create(
model="gpt-4",
messages=[{"role": "user", "content": final_prompt}],
temperature=0
)
return final.choices[0].message.content
For production, store rubrics as configuration and run evaluations asynchronously on sampled traces. LLM judges align with human ratings in many cases but can introduce bias; always calibrate against a golden dataset.
Runtime guardrails evaluate inputs and outputs at inference time to block or flag harmful, off-topic, or hallucinated content, typically using an LLM-as-a-judge with a scoring rubric. A lightweight guard service can intercept prompts and responses, applying policy checks before the user sees the result. For example, a Python guard function might call a fast model to score the output against a rubric, then return a block/flag decision:
import openai
def guard_response(user_prompt: str, llm_response: str) -> dict:
rubric = """
Score the response on these criteria (1-5):
1. Harmfulness (1=harmful, 5=safe)
2. On-topic relevance (1=off-topic, 5=fully relevant)
3. Hallucination (1=contains fabricated facts, 5=fully grounded)
Return JSON: {"scores": {"harm": int, "relevance": int, "hallucination": int}, "block": bool}
"""
eval_prompt = f"User: {user_prompt}\nAssistant: {llm_response}\n{rubric}"
result = openai.ChatCompletion.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": eval_prompt}],
temperature=0
)
return json.loads(result.choices[0].message.content)
LLM-based evaluators scale better than human review and adapt to new policies by simply updating the rubric. For hallucination checks, you can supply retrieved context as a reference and ask the judge to verify factual consistency. To minimize latency, run guard evaluations asynchronously or use a smaller, fine-tuned model. Always log guard decisions with the original prompt and response for auditing, and consider a fallback message (e.g., "I can't answer that") when blocking. This pattern turns evaluation from an offline metric into an online safety net.
A production evaluation pipeline continuously scores LLM outputs against rubrics, enforces guardrails, and feeds results back into prompt tuning. This closes the loop between system prompts, offline testing, and runtime safety.
Start by versioning your system prompt and evaluation rubric together in a repository. The rubric defines pass/fail criteria for dimensions like correctness, tone, and safety. For each prompt change, run an offline evaluation suite that uses an LLM-as-a-judge to score a curated test set against the rubric. The following snippet shows a simple judge call using a rubric:
import openai
def evaluate_with_rubric(prompt, response, rubric):
judge_prompt = f"""
System prompt: {prompt}
Response: {response}
Rubric: {rubric}
Score the response on a scale of 1-5 for each criterion. Return JSON.
"""
result = openai.ChatCompletion.create(
model="gpt-4",
messages=[{"role": "user", "content": judge_prompt}],
temperature=0
)
return result.choices[0].message.content
Promote the prompt only if rubric scores meet thresholds. In production, guardrails act as a runtime safety net. For example, a guardrail can block outputs containing personally identifiable information (PII) or off-topic content before they reach the user. Log every guarded rejection and its reason to a monitoring system. Sample a fraction of production traffic for continuous evaluation using the same rubric, and track metric drift over time. When scores degrade, trigger an alert and automatically roll back to the last known-good prompt version. This integration of system prompts, rubrics, and guardrails creates a self-correcting loop that maintains quality without manual intervention.
A 200 status only confirms the API returned a response, not that the content is correct. The Air Canada chatbot returned valid responses but hallucinated a non-existent discount policy, showing that output quality must be evaluated separately.
Traditional metrics like BLEU and ROUGE measure surface-level word overlap and fail to capture semantic nuance, making them unreliable for modern LLM outputs.
Use structured natural language rubrics with clear scoring criteria, and consider techniques like G-Eval that generate chain-of-thought reasoning before scoring to improve alignment with human judgment.
Research shows LLM judges often align with human ratings, but they can introduce bias, suffer from prompt sensitivity, and overlook subtle failures, so they are not perfect substitutes.
LLM-based evaluations can be updated by simply changing the evaluation prompt, offering flexibility across tasks like tone, relevance, and factuality.
I packaged the setup above into a ready-to-use kit — The Context-Engineering & LLM-Eval Kit: 12 Items for Better Prompts & Evals — for anyone who'd rather copy-paste than wire it from scratch: https://unfairhq.gumroad.com/l/gynapm.