{"slug": "three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches", "title": "Three Unrelated Pipelines, One Root Cause: Building a CircleCI Gate That Catches Broken LLM JSON Before It Ships", "summary": "A developer detailed how three unrelated pipelines failed within 48 hours due to a shared root cause: LLM-generated JSON that couldn't be parsed by json.loads. The developer built a CircleCI gate that freezes real model responses as fixtures and runs a pytest suite on every commit touching prompt templates or parsing schemas, catching contract violations before they ship.", "body_md": "Yesterday, three completely unrelated pipelines in my system failed the same way within 48 hours.\n\nOne generates blog articles. One generates local-business outreach demos. One generates client proposals. They don't share a codebase, a prompt template, or an owner. What they share is a single line buried in each of them: `json.loads(response)`\n\n.\n\nAll three broke with a variation of the same error: *\"content generated but JSON parse failed.\"* Not a crash — worse. A silent stall. The model did its job, produced something, and the parser choked on it downstream, after the artifact was already written to disk. By the time anyone noticed, the failure was three layers removed from its cause.\n\nIf you're shipping any product where an LLM's output gets parsed as structured data — and at this point, whose isn't — this is worth thirty minutes of your CI pipeline's time, because it will happen to you too, and it will happen more than once.\n\nThe instinct is to treat \"parse the model's JSON\" as a one-line implementation detail. It isn't. It's a contract between two systems that drift independently:\n\n`null`\n\non some fraction of calls because the model decided ambiguity was better represented that way.None of these are your bugs, exactly. But they become your outage, because your parser has zero tolerance for any of them, and nothing tells you the contract changed until a human notices missing output days later.\n\nThe first move is obvious and it's what I did initially — wrap the `json.loads`\n\ncall, catch the exception, log it, retry once with a stricter re-prompt. Something like:\n\n``` php\nimport json\nimport re\n\ndef parse_llm_json(raw: str) -> dict:\n    # Strip markdown code fences if the model added them\n    match = re.search(r\"```\n\n(?:json)?\\s*(\\{.*\\})\\s*\n\n```\", raw, re.DOTALL)\n    candidate = match.group(1) if match else raw.strip()\n    return json.loads(candidate)\n\ndef generate_with_repair(prompt: str, call_model, max_attempts: int = 2) -> dict:\n    last_error = None\n    for attempt in range(max_attempts):\n        raw = call_model(prompt if attempt == 0 else\n            f\"{prompt}\\n\\nYour previous response failed to parse as JSON \"\n            f\"({last_error}). Return ONLY valid JSON, no prose, no code fences.\")\n        try:\n            return parse_llm_json(raw)\n        except json.JSONDecodeError as e:\n            last_error = str(e)\n    raise ValueError(f\"Failed to get valid JSON after {max_attempts} attempts: {last_error}\")\n```\n\nThis works. It also completely misses the point, because I wrote a version of this three separate times, once per pipeline, and the underlying contract violation still wasn't caught until runtime — after real API spend, after a task was already marked in-progress, after the failure had to be discovered rather than prevented.\n\nThe deeper mistake was validating **after** the artifact was generated and stored, instead of before the task was allowed to complete. Downstream validation finds the bug. It doesn't stop three independent teams (or three independent pipelines, if you're a solo dev) from rediscovering it separately.\n\nThe actual fix wasn't a better try/except. It was moving the check to a layer where a bad contract gets caught *before* it ships, on every commit that touches a prompt template or a parsing schema — not after a customer-facing task fails.\n\nStep one: freeze a fixture set of real model responses, both the ones that parsed fine and the malformed ones that caused the actual incidents. Keep them as JSONL, one response per line, alongside an expected outcome:\n\n```\n{\"raw\": \"{\\\"title\\\": \\\"Post\\\", \\\"tags\\\": [\\\"a\\\"]}\", \"should_parse\": true}\n{\"raw\": \"Sure, here's the JSON:\\n```\n\njson\\n{\\\"title\\\": \\\"Post\\\"}\\n\n\n```\", \"should_parse\": true}\n{\"raw\": \"{\\\"title\\\": \\\"Post\\\", \\\"tags\\\": [\\\"a\\\",]}\", \"should_parse\": false}\n```\n\nStep two: a pytest suite that runs the real parser against every fixture and asserts the outcome matches:\n\n``` python\nimport json\nimport pytest\nfrom pipeline.parsing import parse_llm_json\n\ndef load_fixtures():\n    with open(\"fixtures/llm_responses.jsonl\") as f:\n        return [json.loads(line) for line in f]\n\n@pytest.mark.parametrize(\"case\", load_fixtures())\ndef test_parser_contract(case):\n    if case[\"should_parse\"]:\n        assert parse_llm_json(case[\"raw\"])  # must not raise\n    else:\n        with pytest.raises(json.JSONDecodeError):\n            parse_llm_json(case[\"raw\"])\n```\n\nStep three, the part that actually stopped the repeat: a dedicated CircleCI job gating anything touching prompts or parsing code.\n\n```\nversion: 2.1\n\njobs:\n  llm-contract-test:\n    docker:\n      - image: cimg/python:3.12\n    steps:\n      - checkout\n      - run: pip install -r requirements.txt\n      - run:\n          name: Run LLM output contract tests\n          command: pytest tests/test_llm_contract.py -v\n\nworkflows:\n  version: 2\n  build-and-test:\n    jobs:\n      - llm-contract-test:\n          filters:\n            branches:\n              only: /.*/\n```\n\nEvery time a prompt template, a parser, or a schema changes, this job runs the full fixture set — the good responses and the bad ones I've actually seen in production — in about eight seconds. If a change makes the parser reject a response it used to accept, or accept one it used to (correctly) reject, the build fails before merge, not three pipelines and three days later.\n\nThe fixture file is now append-only: every time a pipeline hits a new malformed-response shape in production, it gets added to `fixtures/llm_responses.jsonl`\n\nas a new `should_parse: false`\n\ncase before the fix ships. That turns every incident into a permanent regression test instead of a one-off patch. Three months in, the fixture set has caught two prompt-template edits that would have silently broken parsing again — both caught in CI, both zero-impact in production.\n\nThe lesson wasn't \"add a try/except.\" It was that an LLM's output shape is an interface with a version history, and the same discipline you'd apply to an external API contract — recorded fixtures, explicit pass/fail cases, a CI gate — applies here too. The parsing bug wasn't three bugs in three pipelines. It was one missing test suite.", "url": "https://wpnews.pro/news/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches", "canonical_source": "https://dev.to/mukesh_13/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches-broken-llm-json-2dj5", "published_at": "2026-08-28 19:04:59+00:00", "updated_at": "2026-08-28 19:20:53.075132+00:00", "lang": "en", "topics": ["developer-tools", "large-language-models", "mlops"], "entities": ["CircleCI", "pytest", "json.loads"], "alternates": {"html": "https://wpnews.pro/news/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches", "markdown": "https://wpnews.pro/news/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches.md", "text": "https://wpnews.pro/news/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches.txt", "jsonld": "https://wpnews.pro/news/three-unrelated-pipelines-one-root-cause-building-a-circleci-gate-that-catches.jsonld"}}