{"slug": "ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms", "title": "Ollama Structured Outputs in Practice — Getting Type-Safe JSON from Local LLMs with Pydantic", "summary": "Ollama 0.3.0 introduced a format parameter that enforces structured JSON output from local LLMs via constrained decoding, eliminating parsing failures caused by markdown wrappers or extraneous text. Tests with Gemma4 showed a 6.4x speed improvement (5 seconds vs 32 seconds) and reliable JSON parsing when using the format parameter. Developers can integrate Pydantic models to automatically generate JSON schemas and validate outputs.", "body_md": "`json.loads(response)`\n\nfails at a certain point. You told the model \"return JSON only,\" but it added a\n\n``` json markdown code fence around everything. A quick regex strips it — until that regex hits an edge case, and that edge case blows up in production.\n\nSince Ollama 0.3.0, passing a JSON schema to the `format`\n\nparameter eliminates this problem at the root. The model's inference itself is constrained by the schema, so no code fences, no explanatory text, no mid-thought artifacts. Just parseable JSON.\n\nI ran these tests locally with Gemma4 and Ollama 0.30.7 to see how well it holds up in practice.\n\nThe most common problem when running Ollama locally — without a cloud LLM API — is JSON parsing. Two reasons.\n\nFirst, text generation models are trained toward \"natural text.\" Even if you ask for JSON only, they'll often wrap it in\n\njson ...\n\n` blocks or prepend \"Of course! Here is the JSON you requested:\" style text. Here's what I reproduced directly:`\n\n```\njson\nInput: 'Give me 3 Python tips as JSON with keys: tips (array), difficulty (1-5)'\nModel output (no format parameter):\n\n``` json\n{\n  \"tips\": [\n    \"Master the fundamentals first...\",\n    ...\n  ]\n}\n```\n\nJSON parse: FAILED\n\n```\nPython's `json.loads()` can't handle the markdown wrapper. The \"JSON only\" instruction is unreliable in production.\n\nSecond, speed. I measured the same query both ways: 32 seconds without structured output, 5 seconds with it. More on why below.\n\n## How the Ollama format Parameter Works\n\nOllama's `/api/generate` endpoint has a `format` field. Pass a JSON schema object and Ollama applies **constrained decoding** during inference.\n```\n\npython\n\nimport json\n\nimport urllib.request\n\ndef ollama_structured(prompt, schema, model=\"gemma4:e4b\"):\n\npayload = {\n\n\"model\": model,\n\n\"prompt\": prompt,\n\n\"format\": schema, # ← pass JSON schema object directly\n\n\"stream\": False,\n\n\"options\": {\"temperature\": 0}\n\n}\n\ndata = json.dumps(payload).encode()\n\nreq = urllib.request.Request(\n\n\"[http://localhost:11434/api/generate](http://localhost:11434/api/generate)\",\n\ndata=data,\n\nheaders={\"Content-Type\": \"application/json\"}\n\n)\n\nwith urllib.request.urlopen(req, timeout=60) as resp:\n\nresult = json.loads(resp.read())\n\nreturn result[\"response\"]\n\n```\nConstrained decoding sets the probability of any token that would violate the schema to zero at each generation step. So even if the model \"wants\" to generate a markdown fence, the schema makes it physically impossible. That's also where the speed gain comes from — the model doesn't waste tokens on formatting decisions.\n\nHere are the measured numbers:\n```\n\nbash\n\nWithout structured output:\n\nRaw (first 200 chars):\n\n``` json\\n{\\n \"tips\": [\"Master the fundamentals first...\n\nTime: 31.84s\n\nJSON parse: FAILED (markdown wrapper)\n\nWith structured output:\n\nStructured: {\"tips\": [\"Understand the concept of indentation...\", ...], \"difficulty\": 2}\n\nTime: 4.99s\n\nJSON parse: SUCCESS\n\n```\n6.4x difference. Local LLMs are already slow, and adding unreliable parsing on top makes the whole pipeline feel worse.\n\n## Wiring Pydantic Models\n\nWriting JSON schema objects by hand is tedious. With Pydantic models, `model_json_schema()` generates the schema automatically.\n\n``` python\nfrom pydantic import BaseModel\nfrom typing import List, Dict, Any, Literal\n\nclass CodeReview(BaseModel):\n    severity: str  # \"critical\", \"warning\", \"info\"\n    file: str\n    line: int\n    message: str\n    suggestion: str\n\nclass ReviewResult(BaseModel):\n    total_issues: int\n    critical_count: int\n    reviews: List[CodeReview]\n\n# Pydantic → JSON schema, automatically\nschema = ReviewResult.model_json_schema()\n\nraw = ollama_structured(prompt, schema)\n\n# Parses and validates in one step\nresult = ReviewResult.model_validate_json(raw)\n```\n\n`model_validate_json`\n\nparses the JSON string and runs Pydantic validation simultaneously. If `severity`\n\ngets an integer or `line`\n\ngets a string, it throws `ValidationError`\n\n. Catching that and retrying with a modified prompt is the common pattern in real agents.\n\nActual output from the code review test:\n\n```\n=== Code Review Output ===\nTotal issues: 3\nCritical: 2\n  [CRITICAL] process_user_data:2 - SQL Injection Vulnerability (Direct String Formatting)\n  [CRITICAL] process_user_data:3 - Storing Passwords in Plain Text (Data Leakage)\n  [HIGH] process_user_data:4 - Potential Unused/Incomplete Database Interaction\n```\n\n`total_issues: 3`\n\nand `critical_count: 2`\n\ncome in as integers. `if result.critical_count > 0`\n\nbranches safely.\n\nThe strongest use case for structured output is an agent deciding which tool to call next. You pass the tool list and current situation, and get back a type-safe tool call selection.\n\n``` python\nfrom typing import Literal, Dict, Any\n\nclass ToolCall(BaseModel):\n    tool_name: Literal[\"web_search\", \"read_file\", \"write_file\", \"execute_code\"]\n    parameters: Dict[str, Any]\n    reasoning: str\n\nschema = ToolCall.model_json_schema()\n\nuser_task = \"Find the current Bitcoin price and save it to btc_price.txt\"\nprompt = f\"\"\"You are an AI agent. Decide which tool to call next.\nAvailable tools: web_search, read_file, write_file, execute_code\nTask: {user_task}\nChoose ONE tool call.\"\"\"\n\nraw = ollama_structured(prompt, schema)\ntool_call = ToolCall.model_validate_json(raw)\n\nprint(f\"Tool: {tool_call.tool_name}\")\nprint(f\"Params: {tool_call.parameters}\")\n=== Agent tool dispatch ===\nTool: web_search\nParams: {'query': 'current Bitcoin price'}\nReasoning: The task requires finding real-time information...\n\nDispatch: OK (type-safe)\n```\n\nBecause `tool_name`\n\nis typed as `Literal[\"web_search\", \"read_file\", ...]`\n\n, `tool_call.tool_name`\n\nis always one of those four values. If the model invents a nonexistent tool name, Pydantic throws `ValidationError`\n\n. The `if tool_call.tool_name == \"web_search\"`\n\nbranch is safe to write.\n\nThis is architecturally the same as function calling in cloud APIs. Comparing it with [Claude Agent SDK's Tool Use patterns](https://dev.to/en/blog/en/claude-agent-sdk-tool-use-complete-guide-2026) shows an interesting design difference: cloud LLMs handle tool selection natively at the model level, while local Ollama needs an explicit JSON schema + Pydantic validation layer.\n\nHonestly, it doesn't work perfectly in every case. Testing with Gemma4:e4b (4-bit quantized, 4B parameters), I found a few real constraints.\n\n**Deeply nested schemas.** JSON schemas nested 3+ levels deep (`List[Dict[str, List[BaseModel]]]`\n\n) sometimes return empty arrays at intermediate levels. The 12B model (`gemma4:12b-it-qat`\n\n) reduces this, but doesn't eliminate it. This is a fundamental limitation of the model's context handling.\n\n**Optional field handling.** Fields declared as `Optional[str]`\n\nsometimes get filled with empty string `\"\"`\n\ninstead of `null`\n\n. Pydantic validation passes, but semantics differ. You need `@validator`\n\npost-processing.\n\n**Schema size.** A large Pydantic model's JSON schema can reach hundreds of tokens. That occupies context window space, reducing the room available for the actual prompt. Complex schemas need stronger models.\n\nOnce you've deployed Ollama as an API server (covered in the [Ollama FastAPI production guide](https://dev.to/en/blog/en/ollama-fastapi-production-deployment-guide-2026)), switching models at runtime based on schema complexity becomes a viable optimization.\n\n| Situation | Approach | Why |\n|---|---|---|\n| Simple data extraction (1-2 levels) |\n`format` + `json.loads()`\n|\nFast, no overhead |\n| Type validation needed |\n`format` + Pydantic |\nValidationError catches issues early |\n| Agent tool selection |\n`format` + Pydantic `Literal`\n|\nBlocks invalid tool names |\n| Complex nested schema | Consider larger model | Small local model limitations |\n| Simple text response | No `format`\n|\nAvoid unnecessary constrained decoding overhead |\n\nI think of this as a switch that moves JSON parse reliability from \"unreliable\" to \"near 100%.\" There was a time I was appending \"JSON only please\" to every prompt and hoping for the best. Measuring the actual difference made clear how fragile that approach was.\n\n``` python\nimport json\nimport urllib.request\nfrom typing import List, Optional, Dict, Any\nfrom pydantic import BaseModel\n\ndef ollama_structured(prompt: str, model_cls: type[BaseModel], \n                      model: str = \"gemma4:e4b\") -> BaseModel:\n    \"\"\"\n    Helper that combines Ollama structured output + Pydantic validation.\n    \"\"\"\n    schema = model_cls.model_json_schema()\n    payload = {\n        \"model\": model,\n        \"prompt\": prompt,\n        \"format\": schema,\n        \"stream\": False,\n        \"options\": {\"temperature\": 0}\n    }\n    data = json.dumps(payload).encode()\n    req = urllib.request.Request(\n        \"http://localhost:11434/api/generate\",\n        data=data,\n        headers={\"Content-Type\": \"application/json\"}\n    )\n    with urllib.request.urlopen(req, timeout=120) as resp:\n        result = json.loads(resp.read())\n    return model_cls.model_validate_json(result[\"response\"])\n\n# Usage example\nclass SentimentAnalysis(BaseModel):\n    sentiment: str       # \"positive\", \"negative\", \"neutral\"\n    confidence: float    # 0.0 ~ 1.0\n    key_phrases: List[str]\n\nresult = ollama_structured(\n    \"Analyze sentiment: 'This new MacBook is amazing but too expensive'\",\n    SentimentAnalysis\n)\nprint(f\"Sentiment: {result.sentiment} ({result.confidence:.0%})\")\nprint(f\"Key phrases: {result.key_phrases}\")\n```\n\nThis only covers the simplest cases. A real agent needs a bit more.\n\n**Retry logic.** When Pydantic `ValidationError`\n\nfires, retry with a slightly modified prompt — ideally including the error message. Models often self-correct when they can see why they were wrong.\n\n**Streaming.** With `stream: true`\n\n, you can receive the JSON incrementally as it generates. Pair with a streaming JSON parser like `ijson`\n\nfor memory-efficient handling of large responses.\n\n**Model switching.** Route simple extractions to `gemma4:e4b`\n\n(fast) and complex nested schemas to `gemma4:12b-it-qat`\n\n(accurate) at runtime. [Structuring an entire agent with Pydantic AI](https://dev.to/en/blog/en/pydantic-ai-type-safe-agent-tutorial-2026) shows how to abstract this decision to the framework level.\n\nIf you're already running a Gemma4-based agent locally, adding the `format`\n\nparameter today is a one-line change with a measurable reliability improvement. Especially anywhere in the agent loop where an invalid response immediately causes a downstream error.", "url": "https://wpnews.pro/news/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms", "canonical_source": "https://dev.to/jangwook_kim_e31e7291ad98/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms-with-pydantic-m38", "published_at": "2026-06-17 06:38:48+00:00", "updated_at": "2026-06-17 06:51:26.248087+00:00", "lang": "en", "topics": ["large-language-models", "developer-tools", "ai-tools", "natural-language-processing"], "entities": ["Ollama", "Gemma4", "Pydantic", "Python"], "alternates": {"html": "https://wpnews.pro/news/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms", "markdown": "https://wpnews.pro/news/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms.md", "text": "https://wpnews.pro/news/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms.txt", "jsonld": "https://wpnews.pro/news/ollama-structured-outputs-in-practice-getting-type-safe-json-from-local-llms.jsonld"}}