Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics

How we replaced "looks good to me" with automated evaluation catching 92% of hallucinations before deployment

Three months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say "yeah, that looks right."

Then it hit production.

A customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses.

The post-mortem was brutal: we had zero automated evaluation. Our test process was literally "ask 5 questions, read answers, thumbs up."

Academic benchmarks (MMLU, HellaSwag) don't tell you if your system works for your use case. Production evaluation needs:

┌─────────────┐     ┌──────────────┐     ┌────────────────────┐     ┌──────────────┐
│ Test Cases  │────▶│  LLM Under   │────▶│   Judge Ensemble   │────▶│  Metrics &   │
│ (Golden Set)│     │   Test       │     │  - Faithfulness    │     │  Regression  │
└─────────────┘     └──────────────┘     │  - Instruction F.  │     │  Detection   │
                                          │  - JSON Schema     │     └──────┬───────┘
                                          │  - Custom LLM      │            ▼
                                          └────────────────────┘     ┌──────────────┐
                                                                    │  Dashboard/  │
                                                                    │  PR Comments │
                                                                    └──────────────┘
@dataclass(frozen=True)
class TestCase:
    id: str
    input: dict[str, Any]
    expected: dict[str, Any] | None = None
    tags: list[str] = field(default_factory=list)  # ["edge-case", "long-context"]

@dataclass(frozen=True)
class EvaluationResult:
    test_case_id: str
    judge_name: str
    score: float
    passed: bool
    reasoning: str

class Judge(ABC):
    @abstractmethod
    async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ...

class EvaluationHarness:
    def __init__(self, judges: list[Judge]):
        self.judges = judges

    async def evaluate_all(self, test_cases, generate_fn, concurrency=10):
        ...

RAGAS gives you faithfulness and answer relevance. But production needs more:

class LLMJudge(Judge):
    def __init__(self, name, criteria, model="gpt-4o-mini", few_shot_examples=None):
        self.name = name
        self.criteria = criteria
        self.model = model
        self.few_shot = few_shot_examples or []

    async def evaluate(self, test_case, response):
        client = instructor.from_openai(AsyncOpenAI())

        class Output(BaseModel):
            score: float = Field(ge=0, le=1)
            reasoning: str
            passed: bool

        result = await client.chat.completions.create(
            model=self.model,
            response_model=Output,
            messages=[
                {"role": "system", "content": self._system_prompt()},
                *self._few_shot_messages(),
                {"role": "user", "content": self._build_prompt(test_case, response)},
            ],
            temperature=0.0,
        )
        return EvaluationResult(...)
php
def create_faithfulness_judge() -> LLMJudge:
    return LLMJudge(
        name="faithfulness",
        criteria="""
Evaluate whether the ANSWER is faithful to the CONTEXT.
- Score 1.0: All claims in answer are directly supported by context
- Score 0.5: Some claims unsupported but not contradictory
- Score 0.0: Answer contains claims directly contradicted by context
""",
        threshold=0.8,
        few_shot_examples=[
            {
                "input": {
                    "context": "Company founded in 2019. Revenue $10M in 2023.",
                    "answer": "The company was founded in 2019 and reached $10M revenue in 2023."
                },
                "output": {"score": 1.0, "reasoning": "All claims supported", "passed": True}
            },
            {
                "input": {
                    "context": "Product launched in January 2024.",
                    "answer": "The product launched in March 2024 after extensive beta testing."
                },
                "output": {"score": 0.0, "reasoning": "Contradicts launch date", "passed": False}
            },
        ]
    )

Don't start with 1000 cases. Start with 50 real production cases.

{"id": "support-001", "input": {"question": "How do I reset my password?", "context": "..."}, "expected": {"answer": "Use the 'Forgot Password' link..."}, "tags": ["basic", "auth"]}
{"id": "support-042", "input": {"question": "Why was I charged twice?", "context": "..."}, "expected": null, "tags": ["billing", "edge-case"]}

Stratification matters:

Version your dataset: Git-track it. Every production failure becomes a new test case.

def regression_report(self, baseline: dict[str, float]) -> dict[str, Any]:
    current = self.summary()
    report = {}

    for judge_name, metrics in current.items():
        if judge_name not in baseline:
            continue

        baseline_mean = baseline[judge_name]
        current_mean = metrics["mean_score"]
        diff = current_mean - baseline_mean

        report[judge_name] = {
            "baseline": baseline_mean,
            "current": current_mean,
            "delta": diff,
            "regressed": diff < -0.05,  # 5% drop = regression
            "improved": diff > 0.02,
        }

    return report
name: LLM Evaluation
on:
  pull_request:
    paths: ['prompts/**', 'eval/**']
  schedule: ['0 2 * * *']  # Nightly

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: {python-version: '3.11'}

      - name: Install deps
        run: pip install -e .[dev]

      - name: Run evaluation
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
        run: |
          python -m eval.run_suite --config config.yaml --output results.json

      - name: Check regressions
        run: |
          python -m eval.check_regression --baseline baseline.json --current results.json

      - name: Comment PR
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const results = JSON.parse(fs.readFileSync('results.json'));
            const body = `## LLM Evaluation Results
            | Judge | Pass Rate | Mean Score |
            |-------|-----------|------------|
            ${Object.entries(results.summary).map(([k,v]) => 
              `| ${k} | ${(v.pass_rate*100).toFixed(1)}% | ${v.mean_score.toFixed(3)} |`).join('\n')}
            `;
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body
            });

All MIT licensed, production-hardened:

llm-eval-harness

prompt-registry

eval-dashboard

pip install llm-eval-harness
python
from eval.harness import EvaluationHarness
from eval.judges import create_faithfulness_judge, create_instruction_following_judge
from eval.base import TestCase

cases = [
    TestCase(id="1", input={"question": "..."}, expected={"answer": "..."}, tags=["basic"]),
]

judges = [
    create_faithfulness_judge(),
    create_instruction_following_judge(),
]

harness = EvaluationHarness(judges)
results = await harness.evaluate_all(cases, your_llm_function)
print(harness.summary())

harness.save_baseline("baseline.json")

Evaluation is infrastructure, not afterthought.

Your users don't care about your prompt engineering cleverness. They care that the answer is right. Automated evaluation is how you guarantee that at scale.

Code: github.com/yourname/llm-eval-harness | Discussion: Hacker News | Follow: @yourname