Testing Non-Deterministic LLM Pipelines in CI: A Contract-Based Approach

Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters your test suite. Ask GPT-4 or Claude the same question twice and you can get two different (both correct) answers. Teams shipping LLM-backed features often respond to this by writing almost no tests for the LLM-touching code paths, or by asserting on exact string output and then disabling the test the first time it flakes. Neither is sustainable once the feature is in production and a prompt change or model upgrade can silently break behavior.

The fix isn't a clever assertion library. It's splitting what you're actually testing into three tiers with different determinism guarantees, and mapping each tier to a different CI job.

A contract test never calls a live model. It asserts on the shape of what your pipeline produces, not the content. If your orchestrator expects the LLM to return {title, tags, price_usd, full_content}

, a contract test feeds a canned response through your parsing/validation layer and checks it doesn't throw, that required fields are present, and that types match.

import json
import jsonschema
import pytest

PRODUCT_SCHEMA = {
    "type": "object",
    "required": ["title", "tags", "price_usd", "full_content"],
    "properties": {
        "title": {"type": "string", "minLength": 5},
        "tags": {"type": "array", "minItems": 1},
        "price_usd": {"type": "number", "minimum": 1},
        "full_content": {"type": "string", "minLength": 200},
    },
}

@pytest.mark.parametrize("fixture_file", [
    "fixtures/well_formed.json",
    "fixtures/missing_price.json",
    "fixtures/empty_content.json",
])
def test_pipeline_handles_llm_output(fixture_file):
    payload = json.load(open(fixture_file))
    result = parse_and_validate(payload)  # your own code
    if fixture_file == "fixtures/well_formed.json":
        jsonschema.validate(result, PRODUCT_SCHEMA)
    else:
        assert result.rejected_reason is not None

These fixtures are static JSON files checked into the repo. They run in milliseconds, need no API key, and catch the bug class that actually causes outages: your parser choking on a field the model omitted, a null where you expected a string, a schema drift after a prompt edit. This tier runs on every commit and should block merges.

Between "fake JSON fixture" and "call the real API" sits cassette-based replay: record a real model response once, save it to disk, and replay it on every subsequent CI run instead of hitting the network. This is the same idea as VCR for HTTP tests, applied to LLM calls.

import json, os, hashlib

CASSETTE_DIR = "tests/cassettes"

def llm_call_with_cassette(prompt, real_call_fn):
    key = hashlib.sha256(prompt.encode()).hexdigest()[:16]
    path = f"{CASSETTE_DIR}/{key}.json"
    if os.path.exists(path):
        return json.load(open(path))
    if os.environ.get("CI") and not os.environ.get("RECORD_CASSETTES"):
        raise RuntimeError(f"Missing cassette for prompt hash {key}; run with RECORD_CASSETTES=1 locally")
    response = real_call_fn(prompt)
    json.dump(response, open(path, "w"))
    return response

Cassettes are committed to the repo like any other fixture. When you deliberately change a prompt, you re-record locally with RECORD_CASSETTES=1

, review the diff in the cassette file as part of the PR (this is often where a reviewer catches a regression before it ships), and commit the new cassette. CI never touches the network for these tests, so they're fast and free, but they still exercise your real parsing and business logic against real model output, not a hand-written fixture.

This tier calls the actual model with real API keys and real cost. Its job is to answer one question: has the model's behavior drifted since we last checked, in a way our replay cassettes wouldn't reveal? Because output varies, you can't assert equality. Instead assert on properties: is the response valid JSON, is the schema satisfied, is a semantic similarity score against a reference answer above a threshold (using embeddings, not string match), does token usage stay within a cost budget.

def test_live_semantic_similarity(embed_fn, cosine_sim):
    reference = load_reference_embedding("golden_answer.json")
    live_output = call_real_llm(PROMPT)
    similarity = cosine_sim(embed_fn(live_output["full_content"]), reference)
    assert similarity > 0.80  # loose bound; catches total drift, not phrasing

This is where CircleCI's workflow model earns its keep. Define two workflows in .circleci/config.yml

: one gated on every PR that runs tiers 1 and 2 only, and a scheduled workflow that runs tier 3 nightly (or on demand via an approval job), so a flaky live call never blocks a merge.

version: 2.1

jobs:
  fast-tests:
    docker: [{ image: cimg/python:3.12 }]
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run: pytest tests/test_contract.py tests/test_replay.py -v

  live-smoke-tests:
    docker: [{ image: cimg/python:3.12 }]
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run:
          name: Enforce per-run cost ceiling
          command: python scripts/check_budget.py --max-usd 2.00
      - run: pytest tests/test_live_smoke.py -v --reruns 1

workflows:
  on-pr:
    jobs:
      - fast-tests
  nightly-live-check:
    triggers:
      - schedule:
          cron: "0 6 * * *"
          filters:
            branches: { only: main }
    jobs:
      - live-smoke-tests

Two details matter here. First, --reruns 1

on the live job acknowledges that a single API timeout shouldn't page anyone; a persistent two-run failure is a real signal. Second, the check_budget.py

step runs before the pytest call and estimates cost from a token count on the fixed test prompts, failing the job outright if a prompt change would blow past a dollar ceiling — this catches the scenario where someone widens a prompt's context window and 10x's the nightly bill before it happens, not after the invoice arrives.

Teams that skip this split usually end up with one of two failure modes: PRs blocked for minutes waiting on flaky network calls to a rate-limited API, or tests so loosely asserted they'd pass even if the pipeline returned garbage. Separating contract correctness (tier 1), regression detection against known-good real output (tier 2), and live-model drift detection (tier 3) lets each tier use the assertion style that actually fits its determinism guarantee, and lets CircleCI schedule each at the cadence its cost and speed profile deserves. The PR-blocking path stays fast and free; the expensive, flaky, real-money path runs once a day where a human can triage it without holding up a release.