Three Unrelated Pipelines, One Root Cause: Building a CircleCI Gate That Catches Broken LLM JSON Before It Ships

A developer detailed how three unrelated pipelines failed within 48 hours due to a shared root cause: LLM-generated JSON that couldn't be parsed by json.loads. The developer built a CircleCI gate that freezes real model responses as fixtures and runs a pytest suite on every commit touching prompt templates or parsing schemas, catching contract violations before they ship.

Yesterday, three completely unrelated pipelines in my system failed the same way within 48 hours. One generates blog articles. One generates local-business outreach demos. One generates client proposals. They don't share a codebase, a prompt template, or an owner. What they share is a single line buried in each of them: json.loads response . All three broke with a variation of the same error: "content generated but JSON parse failed." Not a crash — worse. A silent stall. The model did its job, produced something, and the parser choked on it downstream, after the artifact was already written to disk. By the time anyone noticed, the failure was three layers removed from its cause. If you're shipping any product where an LLM's output gets parsed as structured data — and at this point, whose isn't — this is worth thirty minutes of your CI pipeline's time, because it will happen to you too, and it will happen more than once. The instinct is to treat "parse the model's JSON" as a one-line implementation detail. It isn't. It's a contract between two systems that drift independently: null on some fraction of calls because the model decided ambiguity was better represented that way.None of these are your bugs, exactly. But they become your outage, because your parser has zero tolerance for any of them, and nothing tells you the contract changed until a human notices missing output days later. The first move is obvious and it's what I did initially — wrap the json.loads call, catch the exception, log it, retry once with a stricter re-prompt. Something like: php import json import re def parse llm json raw: str - dict: Strip markdown code fences if the model added them match = re.search r" ?:json ?\s \{. \} \s ", raw, re.DOTALL candidate = match.group 1 if match else raw.strip return json.loads candidate def generate with repair prompt: str, call model, max attempts: int = 2 - dict: last error = None for attempt in range max attempts : raw = call model prompt if attempt == 0 else f"{prompt}\n\nYour previous response failed to parse as JSON " f" {last error} . Return ONLY valid JSON, no prose, no code fences." try: return parse llm json raw except json.JSONDecodeError as e: last error = str e raise ValueError f"Failed to get valid JSON after {max attempts} attempts: {last error}" This works. It also completely misses the point, because I wrote a version of this three separate times, once per pipeline, and the underlying contract violation still wasn't caught until runtime — after real API spend, after a task was already marked in-progress, after the failure had to be discovered rather than prevented. The deeper mistake was validating after the artifact was generated and stored, instead of before the task was allowed to complete. Downstream validation finds the bug. It doesn't stop three independent teams or three independent pipelines, if you're a solo dev from rediscovering it separately. The actual fix wasn't a better try/except. It was moving the check to a layer where a bad contract gets caught before it ships, on every commit that touches a prompt template or a parsing schema — not after a customer-facing task fails. Step one: freeze a fixture set of real model responses, both the ones that parsed fine and the malformed ones that caused the actual incidents. Keep them as JSONL, one response per line, alongside an expected outcome: {"raw": "{\"title\": \"Post\", \"tags\": \"a\" }", "should parse": true} {"raw": "Sure, here's the JSON:\n json\n{\"title\": \"Post\"}\n ", "should parse": true} {"raw": "{\"title\": \"Post\", \"tags\": \"a\", }", "should parse": false} Step two: a pytest suite that runs the real parser against every fixture and asserts the outcome matches: python import json import pytest from pipeline.parsing import parse llm json def load fixtures : with open "fixtures/llm responses.jsonl" as f: return json.loads line for line in f @pytest.mark.parametrize "case", load fixtures def test parser contract case : if case "should parse" : assert parse llm json case "raw" must not raise else: with pytest.raises json.JSONDecodeError : parse llm json case "raw" Step three, the part that actually stopped the repeat: a dedicated CircleCI job gating anything touching prompts or parsing code. version: 2.1 jobs: llm-contract-test: docker: - image: cimg/python:3.12 steps: - checkout - run: pip install -r requirements.txt - run: name: Run LLM output contract tests command: pytest tests/test llm contract.py -v workflows: version: 2 build-and-test: jobs: - llm-contract-test: filters: branches: only: /. / Every time a prompt template, a parser, or a schema changes, this job runs the full fixture set — the good responses and the bad ones I've actually seen in production — in about eight seconds. If a change makes the parser reject a response it used to accept, or accept one it used to correctly reject, the build fails before merge, not three pipelines and three days later. The fixture file is now append-only: every time a pipeline hits a new malformed-response shape in production, it gets added to fixtures/llm responses.jsonl as a new should parse: false case before the fix ships. That turns every incident into a permanent regression test instead of a one-off patch. Three months in, the fixture set has caught two prompt-template edits that would have silently broken parsing again — both caught in CI, both zero-impact in production. The lesson wasn't "add a try/except." It was that an LLM's output shape is an interface with a version history, and the same discipline you'd apply to an external API contract — recorded fixtures, explicit pass/fail cases, a CI gate — applies here too. The parsing bug wasn't three bugs in three pipelines. It was one missing test suite.