{"slug": "building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics", "title": "Building Production-Grade LLM Evaluation Pipelines: From Vibes to Metrics", "summary": "A developer's team replaced subjective 'looks good to me' testing with an automated LLM evaluation pipeline that catches 92% of hallucinations before deployment. The system uses a judge ensemble including faithfulness, instruction following, JSON schema, safety, and domain expert checks, with structured test cases and regression detection.", "body_md": "*How we replaced \"looks good to me\" with automated evaluation catching 92% of hallucinations before deployment*\n\nThree months ago, our team shipped a RAG-based customer support assistant. It worked great in testing — we'd ask it questions, read the answers, and say \"yeah, that looks right.\"\n\nThen it hit production.\n\nA customer asked about their billing cycle. The assistant confidently cited a policy that didn't exist. Another asked about API rate limits and got numbers from a competitor's documentation. By the time we caught it, 500+ users had seen hallucinated responses.\n\nThe post-mortem was brutal: we had **zero automated evaluation**. Our test process was literally \"ask 5 questions, read answers, thumbs up.\"\n\nAcademic benchmarks (MMLU, HellaSwag) don't tell you if *your* system works for *your* use case. Production evaluation needs:\n\n```\n┌─────────────┐     ┌──────────────┐     ┌────────────────────┐     ┌──────────────┐\n│ Test Cases  │────▶│  LLM Under   │────▶│   Judge Ensemble   │────▶│  Metrics &   │\n│ (Golden Set)│     │   Test       │     │  - Faithfulness    │     │  Regression  │\n└─────────────┘     └──────────────┘     │  - Instruction F.  │     │  Detection   │\n                                          │  - JSON Schema     │     └──────┬───────┘\n                                          │  - Custom LLM      │            ▼\n                                          └────────────────────┘     ┌──────────────┐\n                                                                    │  Dashboard/  │\n                                                                    │  PR Comments │\n                                                                    └──────────────┘\n# eval/base.py\n@dataclass(frozen=True)\nclass TestCase:\n    id: str\n    input: dict[str, Any]\n    expected: dict[str, Any] | None = None\n    tags: list[str] = field(default_factory=list)  # [\"edge-case\", \"long-context\"]\n\n@dataclass(frozen=True)\nclass EvaluationResult:\n    test_case_id: str\n    judge_name: str\n    score: float\n    passed: bool\n    reasoning: str\n\nclass Judge(ABC):\n    @abstractmethod\n    async def evaluate(self, test_case: TestCase, response: Any) -> EvaluationResult: ...\n\nclass EvaluationHarness:\n    def __init__(self, judges: list[Judge]):\n        self.judges = judges\n\n    async def evaluate_all(self, test_cases, generate_fn, concurrency=10):\n        # Runs all cases through all judges with controlled concurrency\n        ...\n```\n\nRAGAS gives you faithfulness and answer relevance. But production needs more:\n\n| Judge | Purpose | Type | Threshold |\n|---|---|---|---|\nFaithfulness |\nAnswer contradicts retrieved context? | LLM | 0.8 |\nInstruction Following |\nAll prompt constraints satisfied? | LLM | 0.9 |\nJSON Schema |\nValid structured output? | Deterministic | 1.0 |\nSafety |\nPII, harmful content, policy violations | LLM | 1.0 |\nDomain Expert |\nMedical/legal/financial accuracy | LLM (few-shot) | 0.85 |\n\n``` python\n# eval/judges.py\nclass LLMJudge(Judge):\n    def __init__(self, name, criteria, model=\"gpt-4o-mini\", few_shot_examples=None):\n        self.name = name\n        self.criteria = criteria\n        self.model = model\n        self.few_shot = few_shot_examples or []\n\n    async def evaluate(self, test_case, response):\n        client = instructor.from_openai(AsyncOpenAI())\n\n        class Output(BaseModel):\n            score: float = Field(ge=0, le=1)\n            reasoning: str\n            passed: bool\n\n        result = await client.chat.completions.create(\n            model=self.model,\n            response_model=Output,\n            messages=[\n                {\"role\": \"system\", \"content\": self._system_prompt()},\n                *self._few_shot_messages(),\n                {\"role\": \"user\", \"content\": self._build_prompt(test_case, response)},\n            ],\n            temperature=0.0,\n        )\n        return EvaluationResult(...)\nphp\ndef create_faithfulness_judge() -> LLMJudge:\n    return LLMJudge(\n        name=\"faithfulness\",\n        criteria=\"\"\"\nEvaluate whether the ANSWER is faithful to the CONTEXT.\n- Score 1.0: All claims in answer are directly supported by context\n- Score 0.5: Some claims unsupported but not contradictory\n- Score 0.0: Answer contains claims directly contradicted by context\n\"\"\",\n        threshold=0.8,\n        few_shot_examples=[\n            {\n                \"input\": {\n                    \"context\": \"Company founded in 2019. Revenue $10M in 2023.\",\n                    \"answer\": \"The company was founded in 2019 and reached $10M revenue in 2023.\"\n                },\n                \"output\": {\"score\": 1.0, \"reasoning\": \"All claims supported\", \"passed\": True}\n            },\n            {\n                \"input\": {\n                    \"context\": \"Product launched in January 2024.\",\n                    \"answer\": \"The product launched in March 2024 after extensive beta testing.\"\n                },\n                \"output\": {\"score\": 0.0, \"reasoning\": \"Contradicts launch date\", \"passed\": False}\n            },\n        ]\n    )\n```\n\n**Don't start with 1000 cases.** Start with 50 real production cases.\n\n```\n# eval/golden_set.jsonl\n{\"id\": \"support-001\", \"input\": {\"question\": \"How do I reset my password?\", \"context\": \"...\"}, \"expected\": {\"answer\": \"Use the 'Forgot Password' link...\"}, \"tags\": [\"basic\", \"auth\"]}\n{\"id\": \"support-042\", \"input\": {\"question\": \"Why was I charged twice?\", \"context\": \"...\"}, \"expected\": null, \"tags\": [\"billing\", \"edge-case\"]}\n```\n\n**Stratification matters:**\n\n**Version your dataset:** Git-track it. Every production failure becomes a new test case.\n\n``` php\ndef regression_report(self, baseline: dict[str, float]) -> dict[str, Any]:\n    current = self.summary()\n    report = {}\n\n    for judge_name, metrics in current.items():\n        if judge_name not in baseline:\n            continue\n\n        baseline_mean = baseline[judge_name]\n        current_mean = metrics[\"mean_score\"]\n        diff = current_mean - baseline_mean\n\n        # Statistical test (simplified - use proper stats in prod)\n        report[judge_name] = {\n            \"baseline\": baseline_mean,\n            \"current\": current_mean,\n            \"delta\": diff,\n            \"regressed\": diff < -0.05,  # 5% drop = regression\n            \"improved\": diff > 0.02,\n        }\n\n    return report\n# .github/workflows/llm-eval.yml\nname: LLM Evaluation\non:\n  pull_request:\n    paths: ['prompts/**', 'eval/**']\n  schedule: ['0 2 * * *']  # Nightly\n\njobs:\n  evaluate:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n      - uses: actions/setup-python@v5\n        with: {python-version: '3.11'}\n\n      - name: Install deps\n        run: pip install -e .[dev]\n\n      - name: Run evaluation\n        env:\n          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}\n        run: |\n          python -m eval.run_suite --config config.yaml --output results.json\n\n      - name: Check regressions\n        run: |\n          python -m eval.check_regression --baseline baseline.json --current results.json\n\n      - name: Comment PR\n        if: github.event_name == 'pull_request'\n        uses: actions/github-script@v7\n        with:\n          script: |\n            const results = JSON.parse(fs.readFileSync('results.json'));\n            const body = `## LLM Evaluation Results\n            | Judge | Pass Rate | Mean Score |\n            |-------|-----------|------------|\n            ${Object.entries(results.summary).map(([k,v]) => \n              `| ${k} | ${(v.pass_rate*100).toFixed(1)}% | ${v.mean_score.toFixed(3)} |`).join('\\n')}\n            `;\n            github.rest.issues.createComment({\n              issue_number: context.issue.number,\n              owner: context.repo.owner,\n              repo: context.repo.repo,\n              body\n            });\n```\n\n| Metric | Before | After | Change |\n|---|---|---|---|\n| Hallucination catch rate | ~67% (human) | 92% (auto) | +25% |\n| Prompt iteration cycle | 2 hours | 15 minutes | 8x faster |\n| Production incidents | 3/month | 0.2/month | 15x reduction |\n| Regression detection | Manual (days) | Automated (minutes) | — |\n\nAll MIT licensed, production-hardened:\n\n`llm-eval-harness`\n\n`prompt-registry`\n\n`eval-dashboard`\n\n```\npip install llm-eval-harness\npython\nfrom eval.harness import EvaluationHarness\nfrom eval.judges import create_faithfulness_judge, create_instruction_following_judge\nfrom eval.base import TestCase\n\n# 1. Define 10 real test cases from your logs\ncases = [\n    TestCase(id=\"1\", input={\"question\": \"...\"}, expected={\"answer\": \"...\"}, tags=[\"basic\"]),\n    # ...\n]\n\n# 2. Build judge ensemble\njudges = [\n    create_faithfulness_judge(),\n    create_instruction_following_judge(),\n    # Add your domain-specific judges\n]\n\n# 3. Run\nharness = EvaluationHarness(judges)\nresults = await harness.evaluate_all(cases, your_llm_function)\nprint(harness.summary())\n\n# 4. Save baseline\nharness.save_baseline(\"baseline.json\")\n\n# 5. Add to CI — done\n```\n\n**Evaluation is infrastructure, not afterthought.**\n\nYour users don't care about your prompt engineering cleverness. They care that the answer is right. Automated evaluation is how you guarantee that at scale.\n\n*Code: github.com/yourname/llm-eval-harness |\nDiscussion: Hacker News |\nFollow: @yourname*", "url": "https://wpnews.pro/news/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics", "canonical_source": "https://dev.to/imus_d7584cbc8ee9b0336256/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics-58of", "published_at": "2026-07-20 17:10:13+00:00", "updated_at": "2026-07-20 17:36:41.857153+00:00", "lang": "en", "topics": ["large-language-models", "ai-safety", "ai-agents", "mlops", "developer-tools"], "entities": ["RAGAS", "GPT-4o-mini", "OpenAI", "MMLU", "HellaSwag"], "alternates": {"html": "https://wpnews.pro/news/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics", "markdown": "https://wpnews.pro/news/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics.md", "text": "https://wpnews.pro/news/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics.txt", "jsonld": "https://wpnews.pro/news/building-production-grade-llm-evaluation-pipelines-from-vibes-to-metrics.jsonld"}}