Three Unrelated Pipelines, One Root Cause: Building a CircleCI Gate That Catches Broken LLM JSON Before It Ships

Yesterday, three completely unrelated pipelines in my system failed the same way within 48 hours.

One generates blog articles. One generates local-business outreach demos. One generates client proposals. They don't share a codebase, a prompt template, or an owner. What they share is a single line buried in each of them: json.loads(response)

.

All three broke with a variation of the same error: "content generated but JSON parse failed." Not a crash — worse. A silent stall. The model did its job, produced something, and the parser choked on it downstream, after the artifact was already written to disk. By the time anyone noticed, the failure was three layers removed from its cause.

If you're shipping any product where an LLM's output gets parsed as structured data — and at this point, whose isn't — this is worth thirty minutes of your CI pipeline's time, because it will happen to you too, and it will happen more than once.

The instinct is to treat "parse the model's JSON" as a one-line implementation detail. It isn't. It's a contract between two systems that drift independently:

null

on some fraction of calls because the model decided ambiguity was better represented that way.None of these are your bugs, exactly. But they become your outage, because your parser has zero tolerance for any of them, and nothing tells you the contract changed until a human notices missing output days later.

The first move is obvious and it's what I did initially — wrap the json.loads

call, catch the exception, log it, retry once with a stricter re-prompt. Something like:

import json
import re

def parse_llm_json(raw: str) -> dict:
    match = re.search(r"```

(?:json)?\s*(\{.*\})\s*

```", raw, re.DOTALL)
    candidate = match.group(1) if match else raw.strip()
    return json.loads(candidate)

def generate_with_repair(prompt: str, call_model, max_attempts: int = 2) -> dict:
    last_error = None
    for attempt in range(max_attempts):
        raw = call_model(prompt if attempt == 0 else
            f"{prompt}\n\nYour previous response failed to parse as JSON "
            f"({last_error}). Return ONLY valid JSON, no prose, no code fences.")
        try:
            return parse_llm_json(raw)
        except json.JSONDecodeError as e:
            last_error = str(e)
    raise ValueError(f"Failed to get valid JSON after {max_attempts} attempts: {last_error}")

This works. It also completely misses the point, because I wrote a version of this three separate times, once per pipeline, and the underlying contract violation still wasn't caught until runtime — after real API spend, after a task was already marked in-progress, after the failure had to be discovered rather than prevented.

The deeper mistake was validating after the artifact was generated and stored, instead of before the task was allowed to complete. Downstream validation finds the bug. It doesn't stop three independent teams (or three independent pipelines, if you're a solo dev) from rediscovering it separately.

The actual fix wasn't a better try/except. It was moving the check to a layer where a bad contract gets caught before it ships, on every commit that touches a prompt template or a parsing schema — not after a customer-facing task fails.

Step one: freeze a fixture set of real model responses, both the ones that parsed fine and the malformed ones that caused the actual incidents. Keep them as JSONL, one response per line, alongside an expected outcome:

{"raw": "{\"title\": \"Post\", \"tags\": [\"a\"]}", "should_parse": true}
{"raw": "Sure, here's the JSON:\n```

json\n{\"title\": \"Post\"}\n

```", "should_parse": true}
{"raw": "{\"title\": \"Post\", \"tags\": [\"a\",]}", "should_parse": false}

Step two: a pytest suite that runs the real parser against every fixture and asserts the outcome matches:

import json
import pytest
from pipeline.parsing import parse_llm_json

def load_fixtures():
    with open("fixtures/llm_responses.jsonl") as f:
        return [json.loads(line) for line in f]

@pytest.mark.parametrize("case", load_fixtures())
def test_parser_contract(case):
    if case["should_parse"]:
        assert parse_llm_json(case["raw"])  # must not raise
    else:
        with pytest.raises(json.JSONDecodeError):
            parse_llm_json(case["raw"])

Step three, the part that actually stopped the repeat: a dedicated CircleCI job gating anything touching prompts or parsing code.

version: 2.1

jobs:
  llm-contract-test:
    docker:
      - image: cimg/python:3.12
    steps:
      - checkout
      - run: pip install -r requirements.txt
      - run:
          name: Run LLM output contract tests
          command: pytest tests/test_llm_contract.py -v

workflows:
  version: 2
  build-and-test:
    jobs:
      - llm-contract-test:
          filters:
            branches:
              only: /.*/

Every time a prompt template, a parser, or a schema changes, this job runs the full fixture set — the good responses and the bad ones I've actually seen in production — in about eight seconds. If a change makes the parser reject a response it used to accept, or accept one it used to (correctly) reject, the build fails before merge, not three pipelines and three days later.

The fixture file is now append-only: every time a pipeline hits a new malformed-response shape in production, it gets added to fixtures/llm_responses.jsonl

as a new should_parse: false

case before the fix ships. That turns every incident into a permanent regression test instead of a one-off patch. Three months in, the fixture set has caught two prompt-template edits that would have silently broken parsing again — both caught in CI, both zero-impact in production.

The lesson wasn't "add a try/except." It was that an LLM's output shape is an interface with a version history, and the same discipline you'd apply to an external API contract — recorded fixtures, explicit pass/fail cases, a CI gate — applies here too. The parsing bug wasn't three bugs in three pipelines. It was one missing test suite.