JSON Schema Doesn't Prevent AI Hallucinations (And That's Okay)

A developer building ShapeCraft, an open-source structured output library, argues that JSON Schema validation prevents malformed responses but does not guarantee factual accuracy, as demonstrated by an invoice extraction example where the model returns a structurally valid but incorrect value. The developer distinguishes between structural correctness and information trustworthiness, calling for a second verification layer in production AI systems.

A few months ago, if you had asked me whether Structured Outputs solved hallucinations, I probably would have said yes. Today, I wouldn't. Not because Structured Outputs don't work-they absolutely do. But because they solve a different problem . After building ShapeCraft, an open-source structured output library supporting OpenAI, Groq, Ollama GBNF , and Anthropic, I realized something that fundamentally changed how I think about production AI. Structure and correctness are two completely different guarantees. Let's say you're extracting data from an invoice. The invoice says: Invoice Total: $900 Your model returns: { "invoiceTotal": 1200 } Let's validate it. | Validation | Result | |---|---| | Valid JSON | ✅ | | Matches JSON Schema | ✅ | | Required field present | ✅ | | Correct data type | ✅ | | Correct answer | ❌ | The response is structurally perfect. The extracted value is still wrong. JSON Schema didn't fail. It did exactly what it was designed to do. JSON Schema is responsible for ensuring your application receives data in a predictable format. It guarantees things like: That makes applications dramatically easier to build. But it does not guarantee: Those are different problems. I've started thinking about AI validation as two independent layers. Can my application safely consume this response? Examples: JSON Schema solves this extremely well. Can I trust the information? Questions become: This layer is much harder. And I think it's where the next generation of AI tooling needs to evolve. Another interesting thing I learned while building ShapeCraft is that "structured output" isn't implemented the same way everywhere. | Provider | Implementation | Guarantee | |---|---|---| | OpenAI | Server-side schema enforcement | Native | | Groq | Server-side schema enforcement | Native | | Ollama | Grammar-constrained decoding GBNF | Constrained | | Anthropic | Prompt + parsing + retries | Best-effort | Every provider can return valid structured output. But they don't achieve it the same way. Understanding those differences helps you make better architectural decisions. One design decision I made was to expose the guarantee level directly instead of hiding it. js const result = await generate model, schema, prompt ; console.log result.guaranteeLevel ; // native // constrained // best-effort This doesn't tell you whether the answer is correct. It tells you how confident you can be that the response structurally matches the schema. I think that's an important distinction. Structured output solved one of the biggest problems in LLM development. Before it, we spent hours fixing malformed JSON. Today, that problem is largely solved. The next challenge isn't generating valid JSON. It's generating information we can trust. Imagine every response including: That's where I believe production AI is heading. I no longer think of AI validation as a single step. It's two completely different layers. First: Is the response structurally correct? Then: Is the information actually correct? Those are different questions. And understanding that distinction has changed how I design AI systems. I'm curious how you're handling this in production. Do you rely solely on schema validation, or are you adding another verification layer after the model responds? If you're interested in structured generation across multiple providers, ShapeCraft is open source. It supports: With support for: GitHub: https://github.com/aviasoletechnologies/shapecraft https://github.com/aviasoletechnologies/shapecraft npm npm install @aviasole/shapecraft pnpm pnpm add @aviasole/shapecraft Provider SDKs: npm install openai npm install groq-sdk npm install @anthropic-ai/sdk Ollama requires no additional SDK js import { generate, openai } from "@aviasole/shapecraft"; import { z } from "zod"; const schema = z.object { name: z.string , age: z.number , } ; const result = await generate openai { model: "gpt-4.1-mini" } , schema, "Generate a fictional user." ; console.log result.data ; console.log result.guaranteeLevel ;