{"slug": "how-i-built-a-ci-safe-eval-harness-for-structured-llm-output", "title": "How I built a CI-safe eval harness for structured LLM output", "summary": "A developer built llm-contract, an open-source CI-safe eval harness for structured LLM output that enforces schema-first JSON extraction, feeds validation errors back to the model for a single retry, and replays recorded case files in pytest so prompt or model changes must pass the full suite before merging. The developer reports that holding out whole sessions rather than using a random 80/20 split avoids leakage from adjacent frames and typically lowers measured accuracy, which they argue is the honest result.", "body_md": "Most teams evaluate LLMs by reading a few outputs and nodding. That works until it doesn't.\n\nI built `llm-contract` because I needed my own pipeline to be honest about whether a change actually made things better.\n\nThe contract has three parts:\n\n**Schema-first extraction.** Every LLM call returns JSON against a fixed schema. If the model drifts, the pipeline fails fast instead of silently producing garbage.\n\n**Validation feedback retries.** When the schema check fails, the error goes back to the model with the exact violation. It gets one retry. Most of the time it fixes itself; the rest get flagged for human review.\n\n**A CI-safe eval harness.** Recorded case files (input, expected output, metadata) run in pytest. No API calls during the test run. Everything is replayed from recordings. A change to the prompt or the model is a commit that either passes the full suite or it doesn't merge.\n\nThe lesson that took me the longest: how you split your data matters more than your model.\n\nI used to take a random 80/20 split. For time-stamped or session-based data, that leaks adjacent frames into training and makes your eval look perfect while production fails. Holding out whole sessions is the honest split. Temporal validation usually drops your measured accuracy by enough to be uncomfortable. That discomfort is the point.\n\nWeights published without raw data. Metrics published without prose. Code with tests. That's the pattern I'm standardizing on.\n\nFull repo: [https://github.com/Reaver1000/llm-contract](https://github.com/Reaver1000/llm-contract)", "url": "https://wpnews.pro/news/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output", "canonical_source": "https://dev.to/reaver1000/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output-5d21", "published_at": "2026-10-30 14:09:41+00:00", "updated_at": "2026-10-30 14:20:05.760914+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "mlops", "structured-data"], "entities": ["llm-contract", "GitHub", "pytest"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output", "markdown": "https://wpnews.pro/news/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output.md", "text": "https://wpnews.pro/news/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output.txt", "jsonld": "https://wpnews.pro/news/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output.jsonld"}}