Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models

A developer has published a guide to building a robust structured-output pipeline for local LLMs, addressing the common problem of models returning invalid JSON. The pipeline combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries, and includes a copy-pasteable structured_extract() function.

You asked a local model for JSON. You got JSON. You json.loads it and — JSONDecodeError: Expecting value. Because buried in the "JSON" was a code fence, three sentences of "Here is your result:", and a trailing comma no parser will forgive. If you've wired a local LLM into an agent, an ETL job, or a backend endpoint, you've hit this. The naive fix is a regex that strips code fences. That regex works until it doesn't, and "until it doesn't" always lands in production at 2 a.m. This article gives you the real fix: a pipeline that combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries. By the end you'll have a copy-pasteable structured extract you can drop into any local-LLM project. Ollama's format parameter accepts two very different things: So the first rule: pass a real JSON Schema, not the string "json". The Python Ollama client makes this trivial with Pydantic: python from ollama import chat from pydantic import BaseModel class Country BaseModel : name: str capital: str languages: list str response = chat model="qwen2.5:7b", messages= {"role": "user", "content": "Tell me about Canada."} , format=Country.model json schema , country = Country.model validate json response.message.content This is the official recommended pattern and it works on Ollama 0.3.0 or newer. For most straightforward schemas on a 7B-plus model, this alone kills the parse failures. But "most straightforward schemas" hides the real edge cases. Three things still bite you: So the robust design is: constrain when you can, defend when you can't, validate always, retry with feedback. If you can't rely on constrained decoding, you need a parser that survives hostile output. json repair PyPI json-repair is the drop-in upgrade for json.loads — it fixes missing quotes, trailing commas, truncated values, and strips stray prose: python import json repair bad = 'Extracting now: {"users": {"name":"Ada","role":"admin",} ,"ok":true' obj = json repair.loads bad - {'users': {'name': 'Ada', 'role': 'admin'} , 'ok': True} Two gotchas from the library docs: json repair also supports schema/pydantic-guided repair and a strict=True mode that raises instead of repairing. We'll use the gentle default. Never trust the parsed object. Validate it against your contract, and when validation fails, retry with the error fed back to the model — not a blind re-roll. One to three attempts is the right ceiling; beyond that, fail loudly and keep the raw output so you can debug. You can get this for free with instructor: python import instructor from pydantic import BaseModel client = instructor.from provider "ollama/qwen2.5:7b" result = client.chat.completions.create model="qwen2.5:7b", messages= {"role": "user", "content": "Classify this support ticket: ..."} , response model=Ticket, max retries=2, timeout=30.0, TOTAL across retries, important for slow local models But instructor leans on the OpenAI-compatible endpoint, which still depends on the backend honoring the schema. The fully self-contained version below works directly against the Ollama chat API and shows exactly what's happening. python from ollama import chat from pydantic import BaseModel, field validator, ValidationError import json repair import json BACKTICK = chr 96 FENCE = BACKTICK 3 def defensive parse content : cleaned = content.strip if cleaned.startswith FENCE : cleaned = cleaned.split chr 10 , 1 1 drop the opening fence line if cleaned.rstrip .endswith FENCE : cleaned = cleaned.rstrip :-3 drop the closing fence line start = cleaned.find "{" end = cleaned.rfind "}" if start = -1 and end = -1: cleaned = cleaned start:end + 1 keep only the {...} body try: return json.loads cleaned except json.JSONDecodeError: return json repair.loads cleaned last resort def structured extract model cls: type BaseModel , prompt: str, model: str = "qwen2.5:7b", max retries: int = 3, : schema = model cls.model json schema last error = None for attempt in range 1, max retries + 1 : messages = {"role": "system", "content": "Return ONLY JSON matching the schema. No prose, no code fences."}, {"role": "user", "content": prompt}, if attempt 1 and last error: messages.append {"role": "user", "content": f"Your previous output failed validation: {last error}\n" f"Fix it to match the schema exactly."} resp = chat model=model, messages=messages, format=schema, constrained decoding Ollama = 0.3.0 options={"temperature": 0}, try: return model cls.model validate defensive parse resp.message.content except ValidationError, ValueError as e: last error = str e raise RuntimeError f"Structured extraction failed after {max retries} attempts. " f"Last error: {last error}\nRaw: {resp.message.content r}" model validate not model validate json takes the already-parsed object, so a JSON-mode endpoint that slips a fence through still lands in the defensive parser. With constrained decoding on, the fence rarely appears, but defense-in-depth is the whole point. | | Mistake | Symptom | Fix | |---|---|---|---| | 1 | Required fields the text doesn't contain | Fabricated values "competitive" becomes 50000 | Use Optional X = None so absence is valid | | 2 | Deeply nested schemas List Dict str, List Model | Empty intermediate arrays on sub-12B models | Keep nested arrays flat; use a bigger model | | 3 | Optional str comes back as empty string not None | None checks silently fail | field validator normalize empty - None | | 4 | One giant 20-field schema | Lower per-field accuracy | Split into 2-3 sequential 6-7 field calls | | 5 | Trusting model output blindly | Valid-but-wrong-shape JSON ships | Always validate against the contract | | 6 | format="json" instead of a schema | Right shape, wrong keys/types | Pass a full JSON Schema object | | 7 | No retry or infinite retry | Lost data or hangs | Retry with error feedback, max 3, then fail loud | | 8 | Schema description as prompt | Model ignores field meaning | Restate semantics in the prompt prose | | 9 | Rigid schema for generative tasks | Stilted, constrained output | Use a system prompt for generation, schema for extraction | Mistake 3 is the one almost everyone ships. Add the normalizer: python from pydantic import BaseModel, field validator from typing import Optional class Review BaseModel : summary: str sentiment: Optional str = None @field validator "sentiment", mode="before" @classmethod def empty to none cls, v : if isinstance v, str and v.strip == "": return None return v Mistake 1 is subtler and more dangerous than a parse error: json.loads succeeds, validation succeeds, and you store a confidently wrong number. Optional plus None is the only honest signal a field was absent. Structured output stops being a coin flip the moment you stop trusting the model and start enforcing a contract. Constrain what you can, defend what you can't, validate everything, and your agent loop stops dying on malformed JSON.