{"slug": "testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach", "title": "Testing Non-Deterministic LLM Pipelines in CI: A Contract-Based Approach", "summary": "A developer proposes a contract-based approach to testing non-deterministic LLM pipelines in CI, splitting tests into three tiers: contract tests with static fixtures, cassette-based replay of recorded model responses, and live model calls to detect behavioral drift. The method avoids flaky tests by asserting on output shape rather than exact content, and runs fast without API keys for most tiers.", "body_md": "Most CI pipelines assume a function called with the same input twice returns the same output. That assumption breaks the moment an LLM call enters your test suite. Ask GPT-4 or Claude the same question twice and you can get two different (both correct) answers. Teams shipping LLM-backed features often respond to this by writing almost no tests for the LLM-touching code paths, or by asserting on exact string output and then disabling the test the first time it flakes. Neither is sustainable once the feature is in production and a prompt change or model upgrade can silently break behavior.\n\nThe fix isn't a clever assertion library. It's splitting what you're actually testing into three tiers with different determinism guarantees, and mapping each tier to a different CI job.\n\nA contract test never calls a live model. It asserts on the *shape* of what your pipeline produces, not the content. If your orchestrator expects the LLM to return `{title, tags, price_usd, full_content}`\n\n, a contract test feeds a canned response through your parsing/validation layer and checks it doesn't throw, that required fields are present, and that types match.\n\n``` python\n# test_contract.py\nimport json\nimport jsonschema\nimport pytest\n\nPRODUCT_SCHEMA = {\n    \"type\": \"object\",\n    \"required\": [\"title\", \"tags\", \"price_usd\", \"full_content\"],\n    \"properties\": {\n        \"title\": {\"type\": \"string\", \"minLength\": 5},\n        \"tags\": {\"type\": \"array\", \"minItems\": 1},\n        \"price_usd\": {\"type\": \"number\", \"minimum\": 1},\n        \"full_content\": {\"type\": \"string\", \"minLength\": 200},\n    },\n}\n\n@pytest.mark.parametrize(\"fixture_file\", [\n    \"fixtures/well_formed.json\",\n    \"fixtures/missing_price.json\",\n    \"fixtures/empty_content.json\",\n])\ndef test_pipeline_handles_llm_output(fixture_file):\n    payload = json.load(open(fixture_file))\n    result = parse_and_validate(payload)  # your own code\n    if fixture_file == \"fixtures/well_formed.json\":\n        jsonschema.validate(result, PRODUCT_SCHEMA)\n    else:\n        assert result.rejected_reason is not None\n```\n\nThese fixtures are static JSON files checked into the repo. They run in milliseconds, need no API key, and catch the bug class that actually causes outages: your parser choking on a field the model omitted, a null where you expected a string, a schema drift after a prompt edit. This tier runs on every commit and should block merges.\n\nBetween \"fake JSON fixture\" and \"call the real API\" sits cassette-based replay: record a real model response once, save it to disk, and replay it on every subsequent CI run instead of hitting the network. This is the same idea as VCR for HTTP tests, applied to LLM calls.\n\n``` python\n# conftest.py\nimport json, os, hashlib\n\nCASSETTE_DIR = \"tests/cassettes\"\n\ndef llm_call_with_cassette(prompt, real_call_fn):\n    key = hashlib.sha256(prompt.encode()).hexdigest()[:16]\n    path = f\"{CASSETTE_DIR}/{key}.json\"\n    if os.path.exists(path):\n        return json.load(open(path))\n    if os.environ.get(\"CI\") and not os.environ.get(\"RECORD_CASSETTES\"):\n        raise RuntimeError(f\"Missing cassette for prompt hash {key}; run with RECORD_CASSETTES=1 locally\")\n    response = real_call_fn(prompt)\n    json.dump(response, open(path, \"w\"))\n    return response\n```\n\nCassettes are committed to the repo like any other fixture. When you deliberately change a prompt, you re-record locally with `RECORD_CASSETTES=1`\n\n, review the diff in the cassette file as part of the PR (this is often where a reviewer catches a regression before it ships), and commit the new cassette. CI never touches the network for these tests, so they're fast and free, but they still exercise your real parsing and business logic against real model output, not a hand-written fixture.\n\nThis tier calls the actual model with real API keys and real cost. Its job is to answer one question: has the model's *behavior* drifted since we last checked, in a way our replay cassettes wouldn't reveal? Because output varies, you can't assert equality. Instead assert on properties: is the response valid JSON, is the schema satisfied, is a semantic similarity score against a reference answer above a threshold (using embeddings, not string match), does token usage stay within a cost budget.\n\n``` python\ndef test_live_semantic_similarity(embed_fn, cosine_sim):\n    reference = load_reference_embedding(\"golden_answer.json\")\n    live_output = call_real_llm(PROMPT)\n    similarity = cosine_sim(embed_fn(live_output[\"full_content\"]), reference)\n    assert similarity > 0.80  # loose bound; catches total drift, not phrasing\n```\n\nThis is where CircleCI's workflow model earns its keep. Define two workflows in `.circleci/config.yml`\n\n: one gated on every PR that runs tiers 1 and 2 only, and a scheduled workflow that runs tier 3 nightly (or on demand via an approval job), so a flaky live call never blocks a merge.\n\n```\nversion: 2.1\n\njobs:\n  fast-tests:\n    docker: [{ image: cimg/python:3.12 }]\n    steps:\n      - checkout\n      - run: pip install -r requirements.txt\n      - run: pytest tests/test_contract.py tests/test_replay.py -v\n\n  live-smoke-tests:\n    docker: [{ image: cimg/python:3.12 }]\n    steps:\n      - checkout\n      - run: pip install -r requirements.txt\n      - run:\n          name: Enforce per-run cost ceiling\n          command: python scripts/check_budget.py --max-usd 2.00\n      - run: pytest tests/test_live_smoke.py -v --reruns 1\n\nworkflows:\n  on-pr:\n    jobs:\n      - fast-tests\n  nightly-live-check:\n    triggers:\n      - schedule:\n          cron: \"0 6 * * *\"\n          filters:\n            branches: { only: main }\n    jobs:\n      - live-smoke-tests\n```\n\nTwo details matter here. First, `--reruns 1`\n\non the live job acknowledges that a single API timeout shouldn't page anyone; a persistent two-run failure is a real signal. Second, the `check_budget.py`\n\nstep runs *before* the pytest call and estimates cost from a token count on the fixed test prompts, failing the job outright if a prompt change would blow past a dollar ceiling — this catches the scenario where someone widens a prompt's context window and 10x's the nightly bill before it happens, not after the invoice arrives.\n\nTeams that skip this split usually end up with one of two failure modes: PRs blocked for minutes waiting on flaky network calls to a rate-limited API, or tests so loosely asserted they'd pass even if the pipeline returned garbage. Separating contract correctness (tier 1), regression detection against known-good real output (tier 2), and live-model drift detection (tier 3) lets each tier use the assertion style that actually fits its determinism guarantee, and lets CircleCI schedule each at the cadence its cost and speed profile deserves. The PR-blocking path stays fast and free; the expensive, flaky, real-money path runs once a day where a human can triage it without holding up a release.", "url": "https://wpnews.pro/news/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach", "canonical_source": "https://dev.to/mukesh_13/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach-3bjn", "published_at": "2026-07-30 05:16:18+00:00", "updated_at": "2026-07-30 05:32:38.959440+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops", "ai-infrastructure"], "entities": ["GPT-4", "Claude"], "alternates": {"html": "https://wpnews.pro/news/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach", "markdown": "https://wpnews.pro/news/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach.md", "text": "https://wpnews.pro/news/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach.txt", "jsonld": "https://wpnews.pro/news/testing-non-deterministic-llm-pipelines-in-ci-a-contract-based-approach.jsonld"}}