Your LLM returns JSON that crashes json.loads about 1 time in 20. Here is how to handle it

If you call an LLM and ask for JSON, it works in the demo and then fails in production at the worst moment. The usual culprits are a markdown code fence around the JSON, a sentence before it ("Sure! Here is the JSON:"), or a trailing comma.

Here is a small, step-by-step way to deal with all three in plain Python.

import json
import re

def extract_json(text: str):
    text = text.strip()
{% endraw %}
json ...
{% raw %}
 ``` fences
    fence = re.search(r"```

(?:json)?\s*(.*?)

```", text, re.DOTALL)
    if fence:
        text = fence.group(1).strip()
    starts = [i for i in (text.find("{"), text.find("[")) if i != -1]
    if not starts:
        raise ValueError("no JSON found")
    start = min(starts)
    end = max(text.rfind("}"), text.rfind("]"))
    return text[start : end + 1]
python
def parse_llm_json(text: str):
    raw = extract_json(text)
    try:
        return json.loads(raw)
    except json.JSONDecodeError:
        fixed = re.sub(r",\s*([}\]])", r"\1", raw)
        return json.loads(fixed)  # let it raise if still broken

Valid JSON can still have the wrong keys. Check the fields you rely on:

def require(data: dict, schema: dict):
    for key, typ in schema.items():
        if key not in data:
            raise KeyError(f"missing key: {key}")
        if not isinstance(data[key], typ):
            raise TypeError(f"{key} should be {typ.__name__}")
    return data

result = require(parse_llm_json(reply), {"title": str, "score": int})

Feed the exact error back to the model. One retry fixes most failures, and more than two retries usually just burns tokens:

def ask_json(call_llm, prompt, schema, retries=1):
    reply = call_llm(prompt)
    for attempt in range(retries + 1):
        try:
            return require(parse_llm_json(reply), schema)
        except (ValueError, KeyError, TypeError) as e:
            if attempt == retries:
                raise
            reply = call_llm(
                f"{prompt}\n\nYour previous answer was invalid: {e}. "
                "Return only the corrected JSON, no extra text."
            )

Parsing defensively is the fallback. Where the provider supports it, use structured outputs so the model is constrained to your schema in the first place. See OpenAI's Structured Outputs guide and Python's json module docs for the error types you can catch (json.JSONDecodeError carries the line and column).

What is the weirdest broken-JSON output you have gotten from a model? I'd like to add it to the test cases.