LLM Evaluation System Prompts Scored Rubrics Runtime Guardrails: A Practical Guide for Production

A developer outlines a practical approach for evaluating LLM outputs in production, combining system prompts, scored rubrics, and runtime guardrails to catch hallucinations and quality failures that standard HTTP status codes miss. The method uses an LLM-as-a-judge to score outputs on dimensions like factual accuracy and relevance, flagging low-quality responses even when the API returns a 200 status code. The guide references the Air Canada chatbot incident as a cautionary example of why output quality evaluation is critical beyond operational health metrics.

Learn how to evaluate LLM outputs in production using system prompts, scored rubrics, and runtime guardrails to prevent hallucinations and ensure quality. TL;DR: To evaluate LLM outputs in production, combine system prompts that define evaluation criteria, scored rubrics using LLM-as-a-judge for dimensions like correctness and relevance, and runtime guardrails that filter or flag unsafe outputs. This approach scales better than human review, adapts via prompt changes, and catches failures that status codes miss, as seen in the Air Canada chatbot case. A 200 status code only confirms the server processed the request—it says nothing about whether the generated text is factual, safe, or useful. The Air Canada chatbot that invented a non-existent bereavement discount returned perfectly valid HTTP responses, yet the hallucinated policy led to a tribunal ruling against the airline. Production evaluation must therefore separate operational health latency, error rates from output quality correctness, relevance, harmlessness . Consider a typical API call that succeeds operationally but fails qualitatively: python import requests response = requests.post "https://api.example.com/v1/chat", json={"model": "gpt-4o", "messages": {"role": "user", "content": "What is Air Canada's bereavement policy?"} }, headers={"Authorization": "Bearer $KEY"} print response.status code 200 print response.json "choices" 0 "message" "content" Output: "Air Canada offers full refunds for bereavement-related cancellations..." A 200 status code and a well-formed JSON body mask a completely fabricated policy. To catch this, you need a separate evaluation layer that scores the output against a rubric. LLM-as-a-judge is a common approach, using a second model to assess the primary output on dimensions like factual accuracy. eval prompt = """ You are an evaluator. Score the following response on factual accuracy from 1 completely fabricated to 5 fully accurate . Response: "{response}" Score: """ score = llm eval eval prompt.format response=chatbot output if int score < 4: alert ops team chatbot output, score This evaluation layer runs alongside every user-facing response, flagging low-quality outputs even when the system returns 200. Without it, you are measuring uptime while your model quietly erodes trust. A system prompt for evaluation must explicitly define the LLM's role as an impartial judge, specify the exact output schema e.g., JSON with a score and reasoning , and embed a detailed scoring rubric to ensure consistent, measurable assessments across all runs. Without this, LLM-as-a-judge outputs drift, undermining reliability. Start by framing the evaluator's identity and task boundaries. Then provide a structured rubric with clear, mutually exclusive levels. For example, a relevance rubric might define: 1 completely off-topic , 2 tangential , 3 partially relevant , 4 mostly relevant , 5 perfectly on-point . The prompt must also mandate a strict output format to enable automated parsing. This approach reduces prompt sensitivity, a known failure mode where small wording changes cause large score variations. Here's a minimal system prompt for an LLM-as-a-judge evaluating answer relevance: You are an impartial evaluation agent. Your task is to score the relevance of a generated answer to a given question. Rubric: 1 - Completely irrelevant, does not address the question. 2 - Tangentially related but misses the core intent. 3 - Partially relevant, addresses some aspects but includes off-topic content. 4 - Mostly relevant, directly addresses the question with minor digressions. 5 - Perfectly relevant, concise and fully on-topic. Output format: Return ONLY a valid JSON object with keys "score" integer and "reasoning" string . In code, you'd pass this system prompt alongside the user message containing the question and answer to evaluate. For example, using the OpenAI Python client: python import openai response = openai.ChatCompletion.create model="gpt-4", messages= {"role": "system", "content": system prompt}, {"role": "user", "content": f"Question: {question}\nAnswer: {answer}"} , temperature=0.0 deterministic scoring Always set temperature to 0 for evaluation to maximize reproducibility. Version your system prompts in a prompt registry and run calibration tests against human-annotated samples to detect bias or inconsistency before production use. LLM-as-a-Judge uses natural language rubrics to score outputs on dimensions like correctness, relevance, and tone; G-Eval chains evaluation steps to improve reliability. This approach replaces brittle string-matching with semantic assessment that scales across tasks and can be updated by simply changing the prompt. A scored rubric defines criteria and a rating scale e.g., 1–5 in plain language. The judge LLM receives the original query, the generated response, and the rubric, then returns a score with justification. For a customer support bot, a correctness rubric might read: "Score 5 if the answer is factually accurate and fully addresses the question; 1 if it contains hallucinated information." Relevance and tone rubrics follow the same pattern. G-Eval extends this by first asking the LLM to generate detailed evaluation steps from the rubric, then using those steps to produce the final score. This chain-of-thought style reduces prompt sensitivity and yields more consistent ratings. The following example uses a simple Python function to call an LLM with a rubric, then a G-Eval style two-step chain: python import openai def score with rubric query, response, rubric : prompt = f"""Evaluate the response based on the rubric. Query: {query} Response: {response} Rubric: {rubric} Provide a score 1-5 and a brief justification.""" result = openai.chat.completions.create model="gpt-4", messages= {"role": "user", "content": prompt} , temperature=0 return result.choices 0 .message.content python def g eval score query, response, rubric : Step 1: Generate evaluation steps step prompt = f"""Given the rubric, produce a numbered list of evaluation steps. Rubric: {rubric}""" steps = openai.chat.completions.create model="gpt-4", messages= {"role": "user", "content": step prompt} , temperature=0 .choices 0 .message.content Step 2: Score using generated steps final prompt = f"""Evaluate the response using the steps below. Query: {query} Response: {response} Steps: {steps} Provide a score 1-5 and justification.""" final = openai.chat.completions.create model="gpt-4", messages= {"role": "user", "content": final prompt} , temperature=0 return final.choices 0 .message.content For production, store rubrics as configuration and run evaluations asynchronously on sampled traces. LLM judges align with human ratings in many cases but can introduce bias; always calibrate against a golden dataset. Runtime guardrails evaluate inputs and outputs at inference time to block or flag harmful, off-topic, or hallucinated content, typically using an LLM-as-a-judge with a scoring rubric. A lightweight guard service can intercept prompts and responses, applying policy checks before the user sees the result. For example, a Python guard function might call a fast model to score the output against a rubric, then return a block/flag decision: php import openai def guard response user prompt: str, llm response: str - dict: rubric = """ Score the response on these criteria 1-5 : 1. Harmfulness 1=harmful, 5=safe 2. On-topic relevance 1=off-topic, 5=fully relevant 3. Hallucination 1=contains fabricated facts, 5=fully grounded Return JSON: {"scores": {"harm": int, "relevance": int, "hallucination": int}, "block": bool} """ eval prompt = f"User: {user prompt}\nAssistant: {llm response}\n{rubric}" result = openai.ChatCompletion.create model="gpt-4o-mini", messages= {"role": "user", "content": eval prompt} , temperature=0 return json.loads result.choices 0 .message.content LLM-based evaluators scale better than human review and adapt to new policies by simply updating the rubric. For hallucination checks, you can supply retrieved context as a reference and ask the judge to verify factual consistency. To minimize latency, run guard evaluations asynchronously or use a smaller, fine-tuned model. Always log guard decisions with the original prompt and response for auditing, and consider a fallback message e.g., "I can't answer that" when blocking. This pattern turns evaluation from an offline metric into an online safety net. A production evaluation pipeline continuously scores LLM outputs against rubrics, enforces guardrails, and feeds results back into prompt tuning. This closes the loop between system prompts, offline testing, and runtime safety. Start by versioning your system prompt and evaluation rubric together in a repository. The rubric defines pass/fail criteria for dimensions like correctness, tone, and safety. For each prompt change, run an offline evaluation suite that uses an LLM-as-a-judge to score a curated test set against the rubric. The following snippet shows a simple judge call using a rubric: python import openai def evaluate with rubric prompt, response, rubric : judge prompt = f""" System prompt: {prompt} Response: {response} Rubric: {rubric} Score the response on a scale of 1-5 for each criterion. Return JSON. """ result = openai.ChatCompletion.create model="gpt-4", messages= {"role": "user", "content": judge prompt} , temperature=0 return result.choices 0 .message.content Promote the prompt only if rubric scores meet thresholds. In production, guardrails act as a runtime safety net. For example, a guardrail can block outputs containing personally identifiable information PII or off-topic content before they reach the user. Log every guarded rejection and its reason to a monitoring system. Sample a fraction of production traffic for continuous evaluation using the same rubric, and track metric drift over time. When scores degrade, trigger an alert and automatically roll back to the last known-good prompt version. This integration of system prompts, rubrics, and guardrails creates a self-correcting loop that maintains quality without manual intervention. A 200 status only confirms the API returned a response, not that the content is correct. The Air Canada chatbot returned valid responses but hallucinated a non-existent discount policy, showing that output quality must be evaluated separately. Traditional metrics like BLEU and ROUGE measure surface-level word overlap and fail to capture semantic nuance, making them unreliable for modern LLM outputs. Use structured natural language rubrics with clear scoring criteria, and consider techniques like G-Eval that generate chain-of-thought reasoning before scoring to improve alignment with human judgment. Research shows LLM judges often align with human ratings, but they can introduce bias, suffer from prompt sensitivity, and overlook subtle failures, so they are not perfect substitutes. LLM-based evaluations can be updated by simply changing the evaluation prompt, offering flexibility across tasks like tone, relevance, and factuality. I packaged the setup above into a ready-to-use kit — The Context-Engineering & LLM-Eval Kit: 12 Items for Better Prompts & Evals — for anyone who'd rather copy-paste than wire it from scratch: https://unfairhq.gumroad.com/l/gynapm https://unfairhq.gumroad.com/l/gynapm?utm source=devto&utm medium=article&utm campaign=the-context-engineering-llm-eval-kit-12- .