json.loads(response)
fails at a certain point. You told the model "return JSON only," but it added a
Since Ollama 0.3.0, passing a JSON schema to the `format`
parameter eliminates this problem at the root. The model's inference itself is constrained by the schema, so no code fences, no explanatory text, no mid-thought artifacts. Just parseable JSON.
I ran these tests locally with Gemma4 and Ollama 0.30.7 to see how well it holds up in practice.
The most common problem when running Ollama locally — without a cloud LLM API — is JSON parsing. Two reasons.
First, text generation models are trained toward "natural text." Even if you ask for JSON only, they'll often wrap it in
json ...
` blocks or prepend "Of course! Here is the JSON you requested:" style text. Here's what I reproduced directly:`
json Input: 'Give me 3 Python tips as JSON with keys: tips (array), difficulty (1-5)' Model output (no format parameter):
{
"tips": [
"Master the fundamentals first...",
...
]
}
JSON parse: FAILED
Python's `json.loads()` can't handle the markdown wrapper. The "JSON only" instruction is unreliable in production.
Second, speed. I measured the same query both ways: 32 seconds without structured output, 5 seconds with it. More on why below.
## How the Ollama format Parameter Works
Ollama's `/api/generate` endpoint has a `format` field. Pass a JSON schema object and Ollama applies **constrained decoding** during inference.
python
import json
import urllib.request
def ollama_structured(prompt, schema, model="gemma4:e4b"):
payload = {
"model": model,
"prompt": prompt,
"format": schema, # ← pass JSON schema object directly
"stream": False,
"options": {"temperature": 0}
}
data = json.dumps(payload).encode()
req = urllib.request.Request(
"http://localhost:11434/api/generate",
data=data,
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=60) as resp:
result = json.loads(resp.read())
return result["response"]
Constrained decoding sets the probability of any token that would violate the schema to zero at each generation step. So even if the model "wants" to generate a markdown fence, the schema makes it physically impossible. That's also where the speed gain comes from — the model doesn't waste tokens on formatting decisions.
Here are the measured numbers:
bash
Without structured output:
Raw (first 200 chars):
Time: 31.84s
JSON parse: FAILED (markdown wrapper)
With structured output:
Structured: {"tips": ["Understand the concept of indentation...", ...], "difficulty": 2}
Time: 4.99s
JSON parse: SUCCESS
6.4x difference. Local LLMs are already slow, and adding unreliable parsing on top makes the whole pipeline feel worse.
Wiring Pydantic Models #
Writing JSON schema objects by hand is tedious. With Pydantic models, model_json_schema() generates the schema automatically.
from pydantic import BaseModel
from typing import List, Dict, Any, Literal
class CodeReview(BaseModel):
severity: str # "critical", "warning", "info"
file: str
line: int
message: str
suggestion: str
class ReviewResult(BaseModel):
total_issues: int
critical_count: int
reviews: List[CodeReview]
schema = ReviewResult.model_json_schema()
raw = ollama_structured(prompt, schema)
result = ReviewResult.model_validate_json(raw)
model_validate_json
parses the JSON string and runs Pydantic validation simultaneously. If severity
gets an integer or line
gets a string, it throws ValidationError
. Catching that and retrying with a modified prompt is the common pattern in real agents.
Actual output from the code review test:
=== Code Review Output ===
Total issues: 3
Critical: 2
[CRITICAL] process_user_data:2 - SQL Injection Vulnerability (Direct String Formatting)
[CRITICAL] process_user_data:3 - Storing Passwords in Plain Text (Data Leakage)
[HIGH] process_user_data:4 - Potential Unused/Incomplete Database Interaction
total_issues: 3
and critical_count: 2
come in as integers. if result.critical_count > 0
branches safely.
The strongest use case for structured output is an agent deciding which tool to call next. You pass the tool list and current situation, and get back a type-safe tool call selection.
from typing import Literal, Dict, Any
class ToolCall(BaseModel):
tool_name: Literal["web_search", "read_file", "write_file", "execute_code"]
parameters: Dict[str, Any]
reasoning: str
schema = ToolCall.model_json_schema()
user_task = "Find the current Bitcoin price and save it to btc_price.txt"
prompt = f"""You are an AI agent. Decide which tool to call next.
Available tools: web_search, read_file, write_file, execute_code
Task: {user_task}
Choose ONE tool call."""
raw = ollama_structured(prompt, schema)
tool_call = ToolCall.model_validate_json(raw)
print(f"Tool: {tool_call.tool_name}")
print(f"Params: {tool_call.parameters}")
=== Agent tool dispatch ===
Tool: web_search
Params: {'query': 'current Bitcoin price'}
Reasoning: The task requires finding real-time information...
Dispatch: OK (type-safe)
Because tool_name
is typed as Literal["web_search", "read_file", ...]
, tool_call.tool_name
is always one of those four values. If the model invents a nonexistent tool name, Pydantic throws ValidationError
. The if tool_call.tool_name == "web_search"
branch is safe to write.
This is architecturally the same as function calling in cloud APIs. Comparing it with Claude Agent SDK's Tool Use patterns shows an interesting design difference: cloud LLMs handle tool selection natively at the model level, while local Ollama needs an explicit JSON schema + Pydantic validation layer.
Honestly, it doesn't work perfectly in every case. Testing with Gemma4:e4b (4-bit quantized, 4B parameters), I found a few real constraints.
Deeply nested schemas. JSON schemas nested 3+ levels deep (List[Dict[str, List[BaseModel]]]
) sometimes return empty arrays at intermediate levels. The 12B model (gemma4:12b-it-qat
) reduces this, but doesn't eliminate it. This is a fundamental limitation of the model's context handling.
Optional field handling. Fields declared as Optional[str]
sometimes get filled with empty string ""
instead of null
. Pydantic validation passes, but semantics differ. You need @validator
post-processing.
Schema size. A large Pydantic model's JSON schema can reach hundreds of tokens. That occupies context window space, reducing the room available for the actual prompt. Complex schemas need stronger models.
Once you've deployed Ollama as an API server (covered in the Ollama FastAPI production guide), switching models at runtime based on schema complexity becomes a viable optimization.
| Situation | Approach | Why |
|---|---|---|
| Simple data extraction (1-2 levels) | ||
format + json.loads() |
||
| Fast, no overhead | ||
| Type validation needed | ||
format + Pydantic |
||
| ValidationError catches issues early | ||
| Agent tool selection | ||
format + Pydantic Literal |
||
| Blocks invalid tool names | ||
| Complex nested schema | Consider larger model | Small local model limitations |
| Simple text response | No format |
|
| Avoid unnecessary constrained decoding overhead |
I think of this as a switch that moves JSON parse reliability from "unreliable" to "near 100%." There was a time I was appending "JSON only please" to every prompt and hoping for the best. Measuring the actual difference made clear how fragile that approach was.
import json
import urllib.request
from typing import List, Optional, Dict, Any
from pydantic import BaseModel
def ollama_structured(prompt: str, model_cls: type[BaseModel],
model: str = "gemma4:e4b") -> BaseModel:
"""
Helper that combines Ollama structured output + Pydantic validation.
"""
schema = model_cls.model_json_schema()
payload = {
"model": model,
"prompt": prompt,
"format": schema,
"stream": False,
"options": {"temperature": 0}
}
data = json.dumps(payload).encode()
req = urllib.request.Request(
"http://localhost:11434/api/generate",
data=data,
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=120) as resp:
result = json.loads(resp.read())
return model_cls.model_validate_json(result["response"])
class SentimentAnalysis(BaseModel):
sentiment: str # "positive", "negative", "neutral"
confidence: float # 0.0 ~ 1.0
key_phrases: List[str]
result = ollama_structured(
"Analyze sentiment: 'This new MacBook is amazing but too expensive'",
SentimentAnalysis
)
print(f"Sentiment: {result.sentiment} ({result.confidence:.0%})")
print(f"Key phrases: {result.key_phrases}")
This only covers the simplest cases. A real agent needs a bit more.
Retry logic. When Pydantic ValidationError
fires, retry with a slightly modified prompt — ideally including the error message. Models often self-correct when they can see why they were wrong.
Streaming. With stream: true
, you can receive the JSON incrementally as it generates. Pair with a streaming JSON parser like ijson
for memory-efficient handling of large responses.
Model switching. Route simple extractions to gemma4:e4b
(fast) and complex nested schemas to gemma4:12b-it-qat
(accurate) at runtime. Structuring an entire agent with Pydantic AI shows how to abstract this decision to the framework level.
If you're already running a Gemma4-based agent locally, adding the format
parameter today is a one-line change with a measurable reliability improvement. Especially anywhere in the agent loop where an invalid response immediately causes a downstream error.