Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics

A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble including faithfulness, instruction following, JSON schema, safety, and domain expert checks, with structured test cases and regression detection.

How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right." Then it hit production. A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses. The post-mortem was brutal: we had zero automated evaluation . Our test process was literally "ask 5 questions, read answers, thumbs up." Academic benchmarks MMLU, HellaSwag don't tell you if your system works for your use case. Production evaluation needs: ┌─────────────┐ ┌──────────────┐ ┌────────────────────┐ ┌──────────────┐ │ Test Cases │────▶│ LLM Under │────▶│ Judge Ensemble │────▶│ Metrics & │ │ Golden Set │ │ Test │ │ - Faithfulness │ │ Regression │ └─────────────┘ └──────────────┘ │ - Instruction F. │ │ Detection │ │ - JSON Schema │ └──────┬───────┘ │ - Custom LLM │ ▼ └────────────────────┘ ┌──────────────┐ │ Dashboard/ │ │ PR Comments │ └──────────────┘ eval/base.py @dataclass frozen=True class TestCase: id: str input: dict str, Any expected: dict str, Any | None = None tags: list str = field default factory=list "edge-case", "long-context" @dataclass frozen=True class EvaluationResult: test case id: str judge name: str score: float passed: bool reasoning: str class Judge ABC : @abstractmethod async def evaluate self, test case: TestCase, response: Any - EvaluationResult: ... class EvaluationHarness: def init self, judges: list Judge : self.judges = judges async def evaluate all self, test cases, generate fn, concurrency=10 : Runs all cases through all judges with controlled concurrency ... RAGAS gives you faithfulness and answer relevance. But production needs more: | Judge | Purpose | Type | Threshold | |---|---|---|---| Faithfulness | Answer contradicts retrieved context? | LLM | 0.8 | Instruction Following | All prompt constraints satisfied? | LLM | 0.9 | JSON Schema | Valid structured output? | Deterministic | 1.0 | Safety | PII, harmful content, policy violations | LLM | 1.0 | Domain Expert | Medical/legal/financial accuracy | LLM few-shot | 0.85 | python eval/judges.py class LLMJudge Judge : def init self, name, criteria, model="gpt-4o-mini", few shot examples=None : self.name = name self.criteria = criteria self.model = model self.few shot = few shot examples or async def evaluate self, test case, response : client = instructor.from openai AsyncOpenAI class Output BaseModel : score: float = Field ge=0, le=1 reasoning: str passed: bool result = await client.chat.completions.create model=self.model, response model=Output, messages= {"role": "system", "content": self. system prompt }, self. few shot messages , {"role": "user", "content": self. build prompt test case, response }, , temperature=0.0, return EvaluationResult ... php def create faithfulness judge - LLMJudge: return LLMJudge name="faithfulness", criteria=""" Evaluate whether the ANSWER is faithful to the CONTEXT. - Score 1.0: All claims in answer are directly supported by context - Score 0.5: Some claims unsupported but not contradictory - Score 0.0: Answer contains claims directly contradicted by context """, threshold=0.8, few shot examples= { "input": { "context": "Company founded in 2019. Revenue $10M in 2023.", "answer": "The company was founded in 2019 and reached $10M revenue in 2023." }, "output": {"score": 1.0, "reasoning": "All claims supported", "passed": True} }, { "input": { "context": "Product launched in January 2024.", "answer": "The product launched in March 2024 after extensive beta testing." }, "output": {"score": 0.0, "reasoning": "Contradicts launch date", "passed": False} }, Don't start with 1000 cases. Start with 50 real production cases. eval/golden set.jsonl {"id": "support-001", "input": {"question": "How do I reset my password?", "context": "..."}, "expected": {"answer": "Use the 'Forgot Password' link..."}, "tags": "basic", "auth" } {"id": "support-042", "input": {"question": "Why was I charged twice?", "context": "..."}, "expected": null, "tags": "billing", "edge-case" } Stratification matters: Version your dataset: Git-track it. Every production failure becomes a new test case. php def regression report self, baseline: dict str, float - dict str, Any : current = self.summary report = {} for judge name, metrics in current.items : if judge name not in baseline: continue baseline mean = baseline judge name current mean = metrics "mean score" diff = current mean - baseline mean Statistical test simplified - use proper stats in prod report judge name = { "baseline": baseline mean, "current": current mean, "delta": diff, "regressed": diff < -0.05, 5% drop = regression "improved": diff 0.02, } return report .github/workflows/llm-eval.yml name: LLM Evaluation on: pull request: paths: 'prompts/ ', 'eval/ ' schedule: '0 2 ' Nightly jobs: evaluate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: actions/setup-python@v5 with: {python-version: '3.11'} - name: Install deps run: pip install -e . dev - name: Run evaluation env: OPENAI API KEY: ${{ secrets.OPENAI API KEY }} run: | python -m eval.run suite --config config.yaml --output results.json - name: Check regressions run: | python -m eval.check regression --baseline baseline.json --current results.json - name: Comment PR if: github.event name == 'pull request' uses: actions/github-script@v7 with: script: | const results = JSON.parse fs.readFileSync 'results.json' ; const body = LLM Evaluation Results | Judge | Pass Rate | Mean Score | |-------|-----------|------------| ${Object.entries results.summary .map k,v = | ${k} | ${ v.pass rate 100 .toFixed 1 }% | ${v.mean score.toFixed 3 } | .join '\n' } ; github.rest.issues.createComment { issue number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body } ; | Metric | Before | After | Change | |---|---|---|---| | Hallucination catch rate | ~67% human | 92% auto | +25% | | Prompt iteration cycle | 2 hours | 15 minutes | 8x faster | | Production incidents | 3/month | 0.2/month | 15x reduction | | Regression detection | Manual days | Automated minutes | — | All MIT licensed, production-hardened: llm-eval-harness prompt-registry eval-dashboard pip install llm-eval-harness python from eval.harness import EvaluationHarness from eval.judges import create faithfulness judge, create instruction following judge from eval.base import TestCase 1. Define 10 real test cases from your logs cases = TestCase id="1", input={"question": "..."}, expected={"answer": "..."}, tags= "basic" , ... 2. Build judge ensemble judges = create faithfulness judge , create instruction following judge , Add your domain-specific judges 3. Run harness = EvaluationHarness judges results = await harness.evaluate all cases, your llm function print harness.summary 4. Save baseline harness.save baseline "baseline.json" 5. Add to CI — done Evaluation is infrastructure, not afterthought. Your users don't care about your prompt engineering cleverness. They care that the answer is right. Automated evaluation is how you guarantee that at scale. Code: github.com/yourname/llm-eval-harness | Discussion: Hacker News | Follow: @yourname