{"slug": "your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for", "title": "Your LLM Returns JSON That Isn't JSON: A Robust Structured-Output Pipeline for Local Models", "summary": "A developer has published a guide to building a robust structured-output pipeline for local LLMs, addressing the common problem of models returning invalid JSON. The pipeline combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries, and includes a copy-pasteable structured_extract() function.", "body_md": "You asked a local model for JSON. You got JSON. You json.loads() it and — JSONDecodeError: Expecting value. Because buried in the \"JSON\" was a code fence, three sentences of \"Here is your result:\", and a trailing comma no parser will forgive.\n\nIf you've wired a local LLM into an agent, an ETL job, or a backend endpoint, you've hit this. The naive fix is a regex that strips code fences. That regex works until it doesn't, and \"until it doesn't\" always lands in production at 2 a.m.\n\nThis article gives you the real fix: a pipeline that combines Ollama's schema-constrained decoding with a resilient parser, schema validation, and feedback-driven retries. By the end you'll have a copy-pasteable structured_extract() you can drop into any local-LLM project.\n\nOllama's format parameter accepts two very different things:\n\nSo the first rule: pass a real JSON Schema, not the string \"json\". The Python Ollama client makes this trivial with Pydantic:\n\n``` python\nfrom ollama import chat\nfrom pydantic import BaseModel\n\nclass Country(BaseModel):\n    name: str\n    capital: str\n    languages: list[str]\n\nresponse = chat(\n    model=\"qwen2.5:7b\",\n    messages=[{\"role\": \"user\", \"content\": \"Tell me about Canada.\"}],\n    format=Country.model_json_schema(),\n)\n\ncountry = Country.model_validate_json(response.message.content)\n```\n\nThis is the official recommended pattern and it works on Ollama 0.3.0 or newer. For most straightforward schemas on a 7B-plus model, this alone kills the parse failures.\n\nBut \"most straightforward schemas\" hides the real edge cases. Three things still bite you:\n\nSo the robust design is: constrain when you can, defend when you can't, validate always, retry with feedback.\n\nIf you can't rely on constrained decoding, you need a parser that survives hostile output. json_repair (PyPI json-repair) is the drop-in upgrade for json.loads() — it fixes missing quotes, trailing commas, truncated values, and strips stray prose:\n\n``` python\nimport json_repair\n\nbad = 'Extracting now: {\"users\":[{\"name\":\"Ada\",\"role\":\"admin\",}],\"ok\":true'\nobj = json_repair.loads(bad)\n# -> {'users': [{'name': 'Ada', 'role': 'admin'}], 'ok': True}\n```\n\nTwo gotchas from the library docs:\n\njson_repair also supports schema/pydantic-guided repair and a strict=True mode that raises instead of repairing. We'll use the gentle default.\n\nNever trust the parsed object. Validate it against your contract, and when validation fails, retry with the error fed back to the model — not a blind re-roll. One to three attempts is the right ceiling; beyond that, fail loudly and keep the raw output so you can debug.\n\nYou can get this for free with instructor:\n\n``` python\nimport instructor\nfrom pydantic import BaseModel\n\nclient = instructor.from_provider(\"ollama/qwen2.5:7b\")\n\nresult = client.chat.completions.create(\n    model=\"qwen2.5:7b\",\n    messages=[{\"role\": \"user\", \"content\": \"Classify this support ticket: ...\"}],\n    response_model=Ticket,\n    max_retries=2,\n    timeout=30.0,   # TOTAL across retries, important for slow local models\n)\n```\n\nBut instructor leans on the OpenAI-compatible endpoint, which still depends on the backend honoring the schema. The fully self-contained version below works directly against the Ollama chat API and shows exactly what's happening.\n\n``` python\nfrom ollama import chat\nfrom pydantic import BaseModel, field_validator, ValidationError\nimport json_repair\nimport json\n\nBACKTICK = chr(96)\nFENCE = BACKTICK * 3\n\ndef _defensive_parse(content):\n    cleaned = content.strip()\n    if cleaned.startswith(FENCE):\n        cleaned = cleaned.split(chr(10), 1)[1]   # drop the opening fence line\n        if cleaned.rstrip().endswith(FENCE):\n            cleaned = cleaned.rstrip()[:-3]       # drop the closing fence line\n    start = cleaned.find(\"{\")\n    end = cleaned.rfind(\"}\")\n    if start != -1 and end != -1:\n        cleaned = cleaned[start:end + 1]         # keep only the {...} body\n    try:\n        return json.loads(cleaned)\n    except json.JSONDecodeError:\n        return json_repair.loads(cleaned)        # last resort\n\ndef structured_extract(\n    model_cls: type[BaseModel],\n    prompt: str,\n    model: str = \"qwen2.5:7b\",\n    max_retries: int = 3,\n):\n    schema = model_cls.model_json_schema()\n    last_error = None\n    for attempt in range(1, max_retries + 1):\n        messages = [\n            {\"role\": \"system\",\n             \"content\": \"Return ONLY JSON matching the schema. No prose, no code fences.\"},\n            {\"role\": \"user\", \"content\": prompt},\n        ]\n        if attempt > 1 and last_error:\n            messages.append(\n                {\"role\": \"user\",\n                 \"content\": f\"Your previous output failed validation: {last_error}\\n\"\n                            f\"Fix it to match the schema exactly.\"}\n            )\n        resp = chat(\n            model=model,\n            messages=messages,\n            format=schema,                 # constrained decoding (Ollama >= 0.3.0)\n            options={\"temperature\": 0},\n        )\n        try:\n            return model_cls.model_validate(_defensive_parse(resp.message.content))\n        except (ValidationError, ValueError) as e:\n            last_error = str(e)\n    raise RuntimeError(\n        f\"Structured extraction failed after {max_retries} attempts. \"\n        f\"Last error: {last_error}\\nRaw: {resp.message.content!r}\"\n    )\n```\n\nmodel_validate (not model_validate_json) takes the already-parsed object, so a JSON-mode endpoint that slips a fence through still lands in the defensive parser. With constrained decoding on, the fence rarely appears, but defense-in-depth is the whole point.\n\n| # | Mistake | Symptom | Fix |\n|---|---|---|---|\n| 1 | Required fields the text doesn't contain | Fabricated values (\"competitive\" becomes 50000) | Use Optional[X] = None so absence is valid |\n| 2 | Deeply nested schemas (List[Dict[str, List[Model]]]) | Empty intermediate arrays on sub-12B models | Keep nested arrays flat; use a bigger model |\n| 3 | Optional[str] comes back as empty string not None | None checks silently fail | field_validator normalize empty -> None |\n| 4 | One giant 20-field schema | Lower per-field accuracy | Split into 2-3 sequential 6-7 field calls |\n| 5 | Trusting model output blindly | Valid-but-wrong-shape JSON ships | Always validate against the contract |\n| 6 | format=\"json\" instead of a schema | Right shape, wrong keys/types | Pass a full JSON Schema object |\n| 7 | No retry or infinite retry | Lost data or hangs | Retry with error feedback, max 3, then fail loud |\n| 8 | Schema description as prompt | Model ignores field meaning | Restate semantics in the prompt prose |\n| 9 | Rigid schema for generative tasks | Stilted, constrained output | Use a system prompt for generation, schema for extraction |\n\nMistake 3 is the one almost everyone ships. Add the normalizer:\n\n``` python\nfrom pydantic import BaseModel, field_validator\nfrom typing import Optional\n\nclass Review(BaseModel):\n    summary: str\n    sentiment: Optional[str] = None\n\n    @field_validator(\"sentiment\", mode=\"before\")\n    @classmethod\n    def empty_to_none(cls, v):\n        if isinstance(v, str) and v.strip() == \"\":\n            return None\n        return v\n```\n\nMistake 1 is subtler and more dangerous than a parse error: json.loads succeeds, validation succeeds, and you store a confidently wrong number. Optional plus None is the only honest signal a field was absent.\n\nStructured output stops being a coin flip the moment you stop trusting the model and start enforcing a contract. Constrain what you can, defend what you can't, validate everything, and your agent loop stops dying on malformed JSON.", "url": "https://wpnews.pro/news/your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for", "canonical_source": "https://dev.to/syed_anzar/your-llm-returns-json-that-isnt-json-a-robust-structured-output-pipeline-for-local-models-2pm9", "published_at": "2026-08-27 05:47:13+00:00", "updated_at": "2026-08-27 06:18:06.570159+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "mlops"], "entities": ["Ollama", "Pydantic", "json_repair", "instructor", "Qwen2.5"], "alternates": {"html": "https://wpnews.pro/news/your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for", "markdown": "https://wpnews.pro/news/your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for.md", "text": "https://wpnews.pro/news/your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for.txt", "jsonld": "https://wpnews.pro/news/your-llm-returns-json-that-isn-t-json-a-robust-structured-output-pipeline-for.jsonld"}}