Testing Non-Deterministic LLM Pipelines in CI: A Contract-Based Approach

A developer proposes a contract-based approach to testing non-deterministic LLM pipelines in CI, splitting tests into three tiers: contract tests with static fixtures, cassette-based replay of recorded model responses, and live model calls to detect behavioral drift. The method avoids flaky tests by asserting on output shape rather than exact content, and runs fast without API keys for most tiers.

Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters your test suite. Ask GPT-4 or Claude the same question twice and you can get two different both correct answers. Teams shipping LLM-backed features often respond to this by writing almost no tests for the LLM-touching code paths, or by asserting on exact string output and then disabling the test the first time it flakes. Neither is sustainable once the feature is in production and a prompt change or model upgrade can silently break behavior. The fix isn't a clever assertion library. It's splitting what you're actually testing into three tiers with different determinism guarantees, and mapping each tier to a different CI job. A contract test never calls a live model. It asserts on the shape of what your pipeline produces, not the content. If your orchestrator expects the LLM to return {title, tags, price usd, full content} , a contract test feeds a canned response through your parsing/validation layer and checks it doesn't throw, that required fields are present, and that types match. python test contract.py import json import jsonschema import pytest PRODUCT SCHEMA = { "type": "object", "required": "title", "tags", "price usd", "full content" , "properties": { "title": {"type": "string", "minLength": 5}, "tags": {"type": "array", "minItems": 1}, "price usd": {"type": "number", "minimum": 1}, "full content": {"type": "string", "minLength": 200}, }, } @pytest.mark.parametrize "fixture file", "fixtures/well formed.json", "fixtures/missing price.json", "fixtures/empty content.json", def test pipeline handles llm output fixture file : payload = json.load open fixture file result = parse and validate payload your own code if fixture file == "fixtures/well formed.json": jsonschema.validate result, PRODUCT SCHEMA else: assert result.rejected reason is not None These fixtures are static JSON files checked into the repo. They run in milliseconds, need no API key, and catch the bug class that actually causes outages: your parser choking on a field the model omitted, a null where you expected a string, a schema drift after a prompt edit. This tier runs on every commit and should block merges. Between "fake JSON fixture" and "call the real API" sits cassette-based replay: record a real model response once, save it to disk, and replay it on every subsequent CI run instead of hitting the network. This is the same idea as VCR for HTTP tests, applied to LLM calls. python conftest.py import json, os, hashlib CASSETTE DIR = "tests/cassettes" def llm call with cassette prompt, real call fn : key = hashlib.sha256 prompt.encode .hexdigest :16 path = f"{CASSETTE DIR}/{key}.json" if os.path.exists path : return json.load open path if os.environ.get "CI" and not os.environ.get "RECORD CASSETTES" : raise RuntimeError f"Missing cassette for prompt hash {key}; run with RECORD CASSETTES=1 locally" response = real call fn prompt json.dump response, open path, "w" return response Cassettes are committed to the repo like any other fixture. When you deliberately change a prompt, you re-record locally with RECORD CASSETTES=1 , review the diff in the cassette file as part of the PR this is often where a reviewer catches a regression before it ships , and commit the new cassette. CI never touches the network for these tests, so they're fast and free, but they still exercise your real parsing and business logic against real model output, not a hand-written fixture. This tier calls the actual model with real API keys and real cost. Its job is to answer one question: has the model's behavior drifted since we last checked, in a way our replay cassettes wouldn't reveal? Because output varies, you can't assert equality. Instead assert on properties: is the response valid JSON, is the schema satisfied, is a semantic similarity score against a reference answer above a threshold using embeddings, not string match , does token usage stay within a cost budget. python def test live semantic similarity embed fn, cosine sim : reference = load reference embedding "golden answer.json" live output = call real llm PROMPT similarity = cosine sim embed fn live output "full content" , reference assert similarity 0.80 loose bound; catches total drift, not phrasing This is where CircleCI's workflow model earns its keep. Define two workflows in .circleci/config.yml : one gated on every PR that runs tiers 1 and 2 only, and a scheduled workflow that runs tier 3 nightly or on demand via an approval job , so a flaky live call never blocks a merge. version: 2.1 jobs: fast-tests: docker: { image: cimg/python:3.12 } steps: - checkout - run: pip install -r requirements.txt - run: pytest tests/test contract.py tests/test replay.py -v live-smoke-tests: docker: { image: cimg/python:3.12 } steps: - checkout - run: pip install -r requirements.txt - run: name: Enforce per-run cost ceiling command: python scripts/check budget.py --max-usd 2.00 - run: pytest tests/test live smoke.py -v --reruns 1 workflows: on-pr: jobs: - fast-tests nightly-live-check: triggers: - schedule: cron: "0 6 " filters: branches: { only: main } jobs: - live-smoke-tests Two details matter here. First, --reruns 1 on the live job acknowledges that a single API timeout shouldn't page anyone; a persistent two-run failure is a real signal. Second, the check budget.py step runs before the pytest call and estimates cost from a token count on the fixed test prompts, failing the job outright if a prompt change would blow past a dollar ceiling — this catches the scenario where someone widens a prompt's context window and 10x's the nightly bill before it happens, not after the invoice arrives. Teams that skip this split usually end up with one of two failure modes: PRs blocked for minutes waiting on flaky network calls to a rate-limited API, or tests so loosely asserted they'd pass even if the pipeline returned garbage. Separating contract correctness tier 1 , regression detection against known-good real output tier 2 , and live-model drift detection tier 3 lets each tier use the assertion style that actually fits its determinism guarantee, and lets CircleCI schedule each at the cadence its cost and speed profile deserves. The PR-blocking path stays fast and free; the expensive, flaky, real-money path runs once a day where a human can triage it without holding up a release.