# How I built a CI-safe eval harness for structured LLM output

> Source: <https://dev.to/reaver1000/how-i-built-a-ci-safe-eval-harness-for-structured-llm-output-5d21>
> Published: 2026-10-30 14:09:41+00:00

Most teams evaluate LLMs by reading a few outputs and nodding. That works until it doesn't.

I built `llm-contract` because I needed my own pipeline to be honest about whether a change actually made things better.

The contract has three parts:

**Schema-first extraction.** Every LLM call returns JSON against a fixed schema. If the model drifts, the pipeline fails fast instead of silently producing garbage.

**Validation feedback retries.** When the schema check fails, the error goes back to the model with the exact violation. It gets one retry. Most of the time it fixes itself; the rest get flagged for human review.

**A CI-safe eval harness.** Recorded case files (input, expected output, metadata) run in pytest. No API calls during the test run. Everything is replayed from recordings. A change to the prompt or the model is a commit that either passes the full suite or it doesn't merge.

The lesson that took me the longest: how you split your data matters more than your model.

I used to take a random 80/20 split. For time-stamped or session-based data, that leaks adjacent frames into training and makes your eval look perfect while production fails. Holding out whole sessions is the honest split. Temporal validation usually drops your measured accuracy by enough to be uncomfortable. That discomfort is the point.

Weights published without raw data. Metrics published without prose. Code with tests. That's the pattern I'm standardizing on.

Full repo: [https://github.com/Reaver1000/llm-contract](https://github.com/Reaver1000/llm-contract)