Ollama Structured Outputs in Practice — Getting Type-Safe JSON from Local LLMs with Pydantic

Ollama 0.3.0 introduced a format parameter that enforces structured JSON output from local LLMs via constrained decoding, eliminating parsing failures caused by markdown wrappers or extraneous text. Tests with Gemma4 showed a 6.4x speed improvement (5 seconds vs 32 seconds) and reliable JSON parsing when using the format parameter. Developers can integrate Pydantic models to automatically generate JSON schemas and validate outputs.

json.loads response fails at a certain point. You told the model "return JSON only," but it added a json markdown code fence around everything. A quick regex strips it — until that regex hits an edge case, and that edge case blows up in production. Since Ollama 0.3.0, passing a JSON schema to the format parameter eliminates this problem at the root. The model's inference itself is constrained by the schema, so no code fences, no explanatory text, no mid-thought artifacts. Just parseable JSON. I ran these tests locally with Gemma4 and Ollama 0.30.7 to see how well it holds up in practice. The most common problem when running Ollama locally — without a cloud LLM API — is JSON parsing. Two reasons. First, text generation models are trained toward "natural text." Even if you ask for JSON only, they'll often wrap it in json ... blocks or prepend "Of course Here is the JSON you requested:" style text. Here's what I reproduced directly: json Input: 'Give me 3 Python tips as JSON with keys: tips array , difficulty 1-5 ' Model output no format parameter : json { "tips": "Master the fundamentals first...", ... } JSON parse: FAILED Python's json.loads can't handle the markdown wrapper. The "JSON only" instruction is unreliable in production. Second, speed. I measured the same query both ways: 32 seconds without structured output, 5 seconds with it. More on why below. How the Ollama format Parameter Works Ollama's /api/generate endpoint has a format field. Pass a JSON schema object and Ollama applies constrained decoding during inference. python import json import urllib.request def ollama structured prompt, schema, model="gemma4:e4b" : payload = { "model": model, "prompt": prompt, "format": schema, ← pass JSON schema object directly "stream": False, "options": {"temperature": 0} } data = json.dumps payload .encode req = urllib.request.Request " http://localhost:11434/api/generate http://localhost:11434/api/generate ", data=data, headers={"Content-Type": "application/json"} with urllib.request.urlopen req, timeout=60 as resp: result = json.loads resp.read return result "response" Constrained decoding sets the probability of any token that would violate the schema to zero at each generation step. So even if the model "wants" to generate a markdown fence, the schema makes it physically impossible. That's also where the speed gain comes from — the model doesn't waste tokens on formatting decisions. Here are the measured numbers: bash Without structured output: Raw first 200 chars : json\n{\n "tips": "Master the fundamentals first... Time: 31.84s JSON parse: FAILED markdown wrapper With structured output: Structured: {"tips": "Understand the concept of indentation...", ... , "difficulty": 2} Time: 4.99s JSON parse: SUCCESS 6.4x difference. Local LLMs are already slow, and adding unreliable parsing on top makes the whole pipeline feel worse. Wiring Pydantic Models Writing JSON schema objects by hand is tedious. With Pydantic models, model json schema generates the schema automatically. python from pydantic import BaseModel from typing import List, Dict, Any, Literal class CodeReview BaseModel : severity: str "critical", "warning", "info" file: str line: int message: str suggestion: str class ReviewResult BaseModel : total issues: int critical count: int reviews: List CodeReview Pydantic → JSON schema, automatically schema = ReviewResult.model json schema raw = ollama structured prompt, schema Parses and validates in one step result = ReviewResult.model validate json raw model validate json parses the JSON string and runs Pydantic validation simultaneously. If severity gets an integer or line gets a string, it throws ValidationError . Catching that and retrying with a modified prompt is the common pattern in real agents. Actual output from the code review test: === Code Review Output === Total issues: 3 Critical: 2 CRITICAL process user data:2 - SQL Injection Vulnerability Direct String Formatting CRITICAL process user data:3 - Storing Passwords in Plain Text Data Leakage HIGH process user data:4 - Potential Unused/Incomplete Database Interaction total issues: 3 and critical count: 2 come in as integers. if result.critical count 0 branches safely. The strongest use case for structured output is an agent deciding which tool to call next. You pass the tool list and current situation, and get back a type-safe tool call selection. python from typing import Literal, Dict, Any class ToolCall BaseModel : tool name: Literal "web search", "read file", "write file", "execute code" parameters: Dict str, Any reasoning: str schema = ToolCall.model json schema user task = "Find the current Bitcoin price and save it to btc price.txt" prompt = f"""You are an AI agent. Decide which tool to call next. Available tools: web search, read file, write file, execute code Task: {user task} Choose ONE tool call.""" raw = ollama structured prompt, schema tool call = ToolCall.model validate json raw print f"Tool: {tool call.tool name}" print f"Params: {tool call.parameters}" === Agent tool dispatch === Tool: web search Params: {'query': 'current Bitcoin price'} Reasoning: The task requires finding real-time information... Dispatch: OK type-safe Because tool name is typed as Literal "web search", "read file", ... , tool call.tool name is always one of those four values. If the model invents a nonexistent tool name, Pydantic throws ValidationError . The if tool call.tool name == "web search" branch is safe to write. This is architecturally the same as function calling in cloud APIs. Comparing it with Claude Agent SDK's Tool Use patterns https://dev.to/en/blog/en/claude-agent-sdk-tool-use-complete-guide-2026 shows an interesting design difference: cloud LLMs handle tool selection natively at the model level, while local Ollama needs an explicit JSON schema + Pydantic validation layer. Honestly, it doesn't work perfectly in every case. Testing with Gemma4:e4b 4-bit quantized, 4B parameters , I found a few real constraints. Deeply nested schemas. JSON schemas nested 3+ levels deep List Dict str, List BaseModel sometimes return empty arrays at intermediate levels. The 12B model gemma4:12b-it-qat reduces this, but doesn't eliminate it. This is a fundamental limitation of the model's context handling. Optional field handling. Fields declared as Optional str sometimes get filled with empty string "" instead of null . Pydantic validation passes, but semantics differ. You need @validator post-processing. Schema size. A large Pydantic model's JSON schema can reach hundreds of tokens. That occupies context window space, reducing the room available for the actual prompt. Complex schemas need stronger models. Once you've deployed Ollama as an API server covered in the Ollama FastAPI production guide https://dev.to/en/blog/en/ollama-fastapi-production-deployment-guide-2026 , switching models at runtime based on schema complexity becomes a viable optimization. | Situation | Approach | Why | |---|---|---| | Simple data extraction 1-2 levels | format + json.loads | Fast, no overhead | | Type validation needed | format + Pydantic | ValidationError catches issues early | | Agent tool selection | format + Pydantic Literal | Blocks invalid tool names | | Complex nested schema | Consider larger model | Small local model limitations | | Simple text response | No format | Avoid unnecessary constrained decoding overhead | I think of this as a switch that moves JSON parse reliability from "unreliable" to "near 100%." There was a time I was appending "JSON only please" to every prompt and hoping for the best. Measuring the actual difference made clear how fragile that approach was. python import json import urllib.request from typing import List, Optional, Dict, Any from pydantic import BaseModel def ollama structured prompt: str, model cls: type BaseModel , model: str = "gemma4:e4b" - BaseModel: """ Helper that combines Ollama structured output + Pydantic validation. """ schema = model cls.model json schema payload = { "model": model, "prompt": prompt, "format": schema, "stream": False, "options": {"temperature": 0} } data = json.dumps payload .encode req = urllib.request.Request "http://localhost:11434/api/generate", data=data, headers={"Content-Type": "application/json"} with urllib.request.urlopen req, timeout=120 as resp: result = json.loads resp.read return model cls.model validate json result "response" Usage example class SentimentAnalysis BaseModel : sentiment: str "positive", "negative", "neutral" confidence: float 0.0 ~ 1.0 key phrases: List str result = ollama structured "Analyze sentiment: 'This new MacBook is amazing but too expensive'", SentimentAnalysis print f"Sentiment: {result.sentiment} {result.confidence:.0%} " print f"Key phrases: {result.key phrases}" This only covers the simplest cases. A real agent needs a bit more. Retry logic. When Pydantic ValidationError fires, retry with a slightly modified prompt — ideally including the error message. Models often self-correct when they can see why they were wrong. Streaming. With stream: true , you can receive the JSON incrementally as it generates. Pair with a streaming JSON parser like ijson for memory-efficient handling of large responses. Model switching. Route simple extractions to gemma4:e4b fast and complex nested schemas to gemma4:12b-it-qat accurate at runtime. Structuring an entire agent with Pydantic AI https://dev.to/en/blog/en/pydantic-ai-type-safe-agent-tutorial-2026 shows how to abstract this decision to the framework level. If you're already running a Gemma4-based agent locally, adding the format parameter today is a one-line change with a measurable reliability improvement. Especially anywhere in the agent loop where an invalid response immediately causes a downstream error.