# Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics > Source: > Published: 2026-07-20 17:10:13+00:00 *How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment* Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had **zero automated evaluation**. Our test process was literally "ask 5 questions, read answers, thumbs up." Academic benchmarks (MMLU, HellaSwag) don't tell you if *your* system works for *your* use case. Production evaluation needs: ``` ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ (Golden Set)│ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ # eval/base.py @dataclass(frozen=True) class TestCase: id: str input: dict[str, Any] expected: dict[str, Any] | None = None tags: list[str] = field(default_factory=list) # ["edge-case", "long-context"] @dataclass(frozen=True) class EvaluationResult: test_case_id: str judge_name: str score: float passed: bool reasoning: str class Judge(ABC): @abstractmethod async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ... class EvaluationHarness: def __init__(self, judges: list[Judge]): self.judges = judges async def evaluate_all(self, test_cases, generate_fn, concurrency=10): # Runs all cases through all judges with controlled concurrency ... ``` RAGAS gives you faithfulness and answer relevance. But production needs more: | Judge | Purpose | Type | Threshold | |---|---|---|---| Faithfulness | Answer contradicts retrieved context? | LLM | 0.8 | Instruction Following | All prompt constraints satisfied? | LLM | 0.9 | JSON Schema | Valid structured output? | Deterministic | 1.0 | Safety | PII, harmful content, policy violations | LLM | 1.0 | Domain Expert | Medical/legal/financial accuracy | LLM (few-shot) | 0.85 | ``` python # eval/judges.py class LLMJudge(Judge): def __init__(self, name, criteria, model="gpt-4o-mini", few_shot_examples=None): self.name = name self.criteria = criteria self.model = model self.few_shot = few_shot_examples or [] async def evaluate(self, test_case, response): client = instructor.from_openai(AsyncOpenAI()) class Output(BaseModel): score: float = Field(ge=0, le=1) reasoning: str passed: bool result = await client.chat.completions.create( model=self.model, response_model=Output, messages=[ {"role": "system", "content": self._system_prompt()}, *self._few_shot_messages(), {"role": "user", "content": self._build_prompt(test_case, response)}, ], temperature=0.0, ) return EvaluationResult(...) php def create_faithfulness_judge() -> LLMJudge: return LLMJudge( name="faithfulness", criteria=""" Evaluate whether the ANSWER is faithful to the CONTEXT. - Score 1.0: All claims in answer are directly supported by context - Score 0.5: Some claims unsupported but not contradictory - Score 0.0: Answer contains claims directly contradicted by context """, threshold=0.8, few_shot_examples=[ { "input": { "context": "Company founded in 2019. Revenue $10M in 2023.", "answer": "The company was founded in 2019 and reached $10M revenue in 2023." }, "output": {"score": 1.0, "reasoning": "All claims supported", "passed": True} }, { "input": { "context": "Product launched in January 2024.", "answer": "The product launched in March 2024 after extensive beta testing." }, "output": {"score": 0.0, "reasoning": "Contradicts launch date", "passed": False} }, ] ) ``` **Don't start with 1000 cases.** Start with 50 real production cases. ``` # eval/golden_set.jsonl {"id": "support-001", "input": {"question": "How do I reset my password?", "context": "..."}, "expected": {"answer": "Use the 'Forgot Password' link..."}, "tags": ["basic", "auth"]} {"id": "support-042", "input": {"question": "Why was I charged twice?", "context": "..."}, "expected": null, "tags": ["billing", "edge-case"]} ``` **Stratification matters:** **Version your dataset:** Git-track it. Every production failure becomes a new test case. ``` php def regression_report(self, baseline: dict[str, float]) -> dict[str, Any]: current = self.summary() report = {} for judge_name, metrics in current.items(): if judge_name not in baseline: continue baseline_mean = baseline[judge_name] current_mean = metrics["mean_score"] diff = current_mean - baseline_mean # Statistical test (simplified - use proper stats in prod) report[judge_name] = { "baseline": baseline_mean, "current": current_mean, "delta": diff, "regressed": diff < -0.05, # 5% drop = regression "improved": diff > 0.02, } return report # .github/workflows/llm-eval.yml name: LLM Evaluation on: pull_request: paths: ['prompts/**', 'eval/**'] schedule: ['0 2 * * *'] # Nightly jobs: evaluate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: {python-version: '3.11'} - name: Install deps run: pip install -e .[dev] - name: Run evaluation env: OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} run: | python -m eval.run_suite --config config.yaml --output results.json - name: Check regressions run: | python -m eval.check_regression --baseline baseline.json --current results.json - name: Comment PR if: github.event_name == 'pull_request' uses: actions/github-script@v7 with: script: | const results = JSON.parse(fs.readFileSync('results.json')); const body = `## LLM Evaluation Results | Judge | Pass Rate | Mean Score | |-------|-----------|------------| ${Object.entries(results.summary).map(([k,v]) => `| ${k} | ${(v.pass_rate*100).toFixed(1)}% | ${v.mean_score.toFixed(3)} |`).join('\n')} `; github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body }); ``` | Metric | Before | After | Change | |---|---|---|---| | Hallucination catch rate | ~67% (human) | 92% (auto) | +25% | | Prompt iteration cycle | 2 hours | 15 minutes | 8x faster | | Production incidents | 3/month | 0.2/month | 15x reduction | | Regression detection | Manual (days) | Automated (minutes) | — | All MIT licensed, production-hardened: `llm-eval-harness` `prompt-registry` `eval-dashboard` ``` pip install llm-eval-harness python from eval.harness import EvaluationHarness from eval.judges import create_faithfulness_judge, create_instruction_following_judge from eval.base import TestCase # 1. Define 10 real test cases from your logs cases = [ TestCase(id="1", input={"question": "..."}, expected={"answer": "..."}, tags=["basic"]), # ... ] # 2. Build judge ensemble judges = [ create_faithfulness_judge(), create_instruction_following_judge(), # Add your domain-specific judges ] # 3. Run harness = EvaluationHarness(judges) results = await harness.evaluate_all(cases, your_llm_function) print(harness.summary()) # 4. Save baseline harness.save_baseline("baseline.json") # 5. Add to CI — done ``` **Evaluation is infrastructure, not afterthought.** Your users don't care about your prompt engineering cleverness. They care that the answer is right. Automated evaluation is how you guarantee that at scale. *Code: github.com/yourname/llm-eval-harness | Discussion: Hacker News | Follow: @yourname*