Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models

You asked a local model for JSON. You got JSON. You json.loads() it and — JSONDecodeError: Expecting value. Because buried in the "JSON" was a code fence, three sentences of "Here is your result:", and a trailing comma no parser will forgive.

If you've wired a local LLM into an agent, an ETL job, or a backend endpoint, you've hit this. The naive fix is a regex that strips code fences. That regex works until it doesn't, and "until it doesn't" always lands in production at 2 a.m.

This article gives you the real fix: a pipeline that combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries. By the end you'll have a copy-pasteable structured_extract() you can drop into any local-LLM project.

Ollama's format parameter accepts two very different things:

So the first rule: pass a real JSON Schema, not the string "json". The Python Ollama client makes this trivial with Pydantic:

from ollama import chat
from pydantic import BaseModel

class Country(BaseModel):
    name: str
    capital: str
    languages: list[str]

response = chat(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Tell me about Canada."}],
    format=Country.model_json_schema(),
)

country = Country.model_validate_json(response.message.content)

This is the official recommended pattern and it works on Ollama 0.3.0 or newer. For most straightforward schemas on a 7B-plus model, this alone kills the parse failures.

But "most straightforward schemas" hides the real edge cases. Three things still bite you:

So the robust design is: constrain when you can, defend when you can't, validate always, retry with feedback.

If you can't rely on constrained decoding, you need a parser that survives hostile output. json_repair (PyPI json-repair) is the drop-in upgrade for json.loads() — it fixes missing quotes, trailing commas, truncated values, and strips stray prose:

import json_repair

bad = 'Extracting now: {"users":[{"name":"Ada","role":"admin",}],"ok":true'
obj = json_repair.loads(bad)

Two gotchas from the library docs:

json_repair also supports schema/pydantic-guided repair and a strict=True mode that raises instead of repairing. We'll use the gentle default.

Never trust the parsed object. Validate it against your contract, and when validation fails, retry with the error fed back to the model — not a blind re-roll. One to three attempts is the right ceiling; beyond that, fail loudly and keep the raw output so you can debug.

You can get this for free with instructor:

import instructor
from pydantic import BaseModel

client = instructor.from_provider("ollama/qwen2.5:7b")

result = client.chat.completions.create(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Classify this support ticket: ..."}],
    response_model=Ticket,
    max_retries=2,
    timeout=30.0,   # TOTAL across retries, important for slow local models
)

But instructor leans on the OpenAI-compatible endpoint, which still depends on the backend honoring the schema. The fully self-contained version below works directly against the Ollama chat API and shows exactly what's happening.

from ollama import chat
from pydantic import BaseModel, field_validator, ValidationError
import json_repair
import json

BACKTICK = chr(96)
FENCE = BACKTICK * 3

def _defensive_parse(content):
    cleaned = content.strip()
    if cleaned.startswith(FENCE):
        cleaned = cleaned.split(chr(10), 1)[1]   # drop the opening fence line
        if cleaned.rstrip().endswith(FENCE):
            cleaned = cleaned.rstrip()[:-3]       # drop the closing fence line
    start = cleaned.find("{")
    end = cleaned.rfind("}")
    if start != -1 and end != -1:
        cleaned = cleaned[start:end + 1]         # keep only the {...} body
    try:
        return json.loads(cleaned)
    except json.JSONDecodeError:
        return json_repair.loads(cleaned)        # last resort

def structured_extract(
    model_cls: type[BaseModel],
    prompt: str,
    model: str = "qwen2.5:7b",
    max_retries: int = 3,
):
    schema = model_cls.model_json_schema()
    last_error = None
    for attempt in range(1, max_retries + 1):
        messages = [
            {"role": "system",
             "content": "Return ONLY JSON matching the schema. No prose, no code fences."},
            {"role": "user", "content": prompt},
        ]
        if attempt > 1 and last_error:
            messages.append(
                {"role": "user",
                 "content": f"Your previous output failed validation: {last_error}\n"
                            f"Fix it to match the schema exactly."}
            )
        resp = chat(
            model=model,
            messages=messages,
            format=schema,                 # constrained decoding (Ollama >= 0.3.0)
            options={"temperature": 0},
        )
        try:
            return model_cls.model_validate(_defensive_parse(resp.message.content))
        except (ValidationError, ValueError) as e:
            last_error = str(e)
    raise RuntimeError(
        f"Structured extraction failed after {max_retries} attempts. "
        f"Last error: {last_error}\nRaw: {resp.message.content!r}"
    )

model_validate (not model_validate_json) takes the already-parsed object, so a JSON-mode endpoint that slips a fence through still lands in the defensive parser. With constrained decoding on, the fence rarely appears, but defense-in-depth is the whole point.

Mistake 3 is the one almost everyone ships. Add the normalizer:

from pydantic import BaseModel, field_validator
from typing import Optional

class Review(BaseModel):
    summary: str
    sentiment: Optional[str] = None

    @field_validator("sentiment", mode="before")
    @classmethod
    def empty_to_none(cls, v):
        if isinstance(v, str) and v.strip() == "":
            return None
        return v

Mistake 1 is subtler and more dangerous than a parse error: json.loads succeeds, validation succeeds, and you store a confidently wrong number. Optional plus None is the only honest signal a field was absent.

Structured output stops being a coin flip the moment you stop trusting the model and start enforcing a contract. Constrain what you can, defend what you can't, validate everything, and your agent loop stops dying on malformed JSON.