aie_2.2: structured outputs, getting reliable json from an llm

A developer's AI engineering series (aie_2.2) explains how to get reliable JSON from an LLM using structured outputs, framing the model as a new assistant that must return data in a consistent layout for the rest of an application to consume. The piece contrasts structured outputs with function calling and streaming, which are slated for the next installment, and recaps foundational concepts including tokens, statelessness, and context.

Before I introduce anything, I want to take you back to where we left off. In aie 2.0, we went inside an LLM API call. We looked at what you send the messages array and a handful of parameters and what you get back the response object . In aie 2.1, we took those pieces and built what we called a conversation engine, a small program you can chat with in your terminal. If you followed along with the build, then you’ve had a back-and-forth conversation with an LLM, running on your code. What we built there is a chatbot. And a chatbot is the simplest thing you can build with an LLM. You type something, the model writes something back and you read it. Most applications ask much more of a model than that. In a real app, one part among many. It gets handed a task and whatever it produces gets passed along to other parts of the app, to a database, to a screen, to a customer . So its answers have to fit into work that's already happening inside of said application. That is a bit abstract, so I’ll solidify it. Think about what it's like to bring a new assistant into a small business. They're smart, they write well, and they're eager to help. By lunchtime, you've handed them three tasks. Each task comes with something you'd expect from them. If you ask them to fill in a spreadsheet, what do you need from them? The right information, in the right columns, laid out the same way every time. If you ask them about something only you have the records for, like a customer’s order, what should they do? Go and look it up. And if the answer takes a while, what would you like while you wait? Some sign that they’re working on it, so you know something is happening. The model in your app is that new assistant. It's smart, it writes well, and the app hands it tasks. So the app needs the same three things from the model that you'd need from your assistant. It needs answers laid out the same way every time, so the rest of the app can use them. It needs a way for the model to ask for information that only the app's records hold. And when an answer takes a while, it needs to show the person waiting that something is happening. The LLM API has a feature for each one. Structured outputs gets answers in the same layout every time. Function calling lets the model ask your app to look things up. Streaming shows the answer while it's being written. This article is all about the first one, structured outputs. Function calling and streaming get the next article. Before we get into it, let’s refresh a few terms. Every one of these comes back in this article, so I’d like them fresh in your mind before we build on them. A Quick Refresher In aie 1.0, we learned that an LLM is a prediction machine . A Large Language Model LLM is a prediction machine that is trained on a large amount of text and is capable of generating coherent, contextually appropriate language by predicting likely continuations of any input given. The key word is still predict . The model writes its answer one small piece at a time. At every step, it looks at everything it has so far and picks the most likely next piece. Each of those small pieces is called a token . A token is the smallest unit the model works with. Think of it as a chunk of text. “Hello” is one token. “Unbelievable” might be three. We also learned that LLMs have amnesia. The model forgets each call the moment it ends, and every new call arrives as if it’s meeting you for the first time. A system is stateless when each request message is handled completely on its own, with zero memory of any request that came before it. So to make the model "remember" , the app sends the whole conversation back on every call. That conversation is part of the model's context, everything the model can see at the moment it writes a response. In aie 2.0, we pictured all of this as a phone call where the person on the other end has amnesia. Every time you call, they pick up with zero idea who you are. The only way to have a conversation is to read them the full transcript from the beginning, every single time. That transcript is the messages array. The messages array is the list of all that you send to the model on a single API call. It is structured as a sequence of messages, each one labelled with who sent it. Each label is a role. user is whoever is on the other end of your product, like your customer. assistant is the model. Alongside the messages array, you send parameters. Parameters are settings you include with every API call that shape how the model responds. We covered three of them. model picks which version of Claude to use. temperature controls how tightly the model sticks to the most likely next word. At 0, it always picks the most probable one. And max tokens sets a hard ceiling on how long the reply can be, measured in tokens. Finally, what comes back is the response object. We described it as a delivery with a receipt attached. The package is the model’s reply, and the receipt tells you what it cost. - response.content is a list of content blocks, where the model’s reply lives. response.content 0 .text takes the first block and pulls out the words. - response.usage holds the token counts, input tokens for what you sent and output tokens for what the model wrote back. The response object has one more field we'll need in this article. We'll meet it when it comes up. Also, keep that phone call in mind as we go. Structured outputs changes the call in one small way, and we'll come back to it. From a Chatbot to a Real App To see what a real app asks of an LLM, let’s follow one example from the start of this article to the end. Kora Home is a small online shop that sells home goods, things like lamps, throws and ceramic mugs. They want an assistant that answers customer messages. One morning, a customer named Amara sends this. Amara: Hi, this is Amara. My order 4821 was due last week and I’m still waiting. Where is it? For the assistant to handle Amara’s message properly, the app has three jobs to do. - Job one. Turn her message into a record. Kora Home keeps a support log in a database table. Each row holds three things: the customer's name, the order number and what they want. So the app needs three separate, clean values out of Amara's message. Amara , 4821 and order status . - Job two. Find out where order 4821 is. That information is stored in Kora Home’s database, on Kora Home’s computers. - Job three. Show Amara the answer. The reply appears in a chat window on her screen. With only what we learned in aie 2.0, each job has its issue: - For job one, the model replies in free text. The wording changes from call to call, and a program needs the same values in the same places every time. - For job two, the model can only work with what’s in its context. Order 4821 is stored in the shop’s database, outside the model’s context. - For job three, a long answer takes several seconds to write , and Amara looks at an empty chat window the whole time. Here’s how the three features map onto the three jobs. - Structured outputs handles job one. - Function calling handles job two. - Streaming handles job three. In this lesson, we take job one. Jobs two and three are the next article, and Amara's message is what we’ll use in those. This is important because as an AI Engineer, most of your work happens in the space between the model and the rest of your app. The model writes text and your job is getting that text into a shape your code can use, connecting the model to the information it needs, and delivering its answers to people in a way that makes sense to use. These three features are your first tools for that work. Structured Outputs Let’s look at turning Amara’s message into a record. Remember the assistant with the spreadsheet, Kora Home's support log is a table with three columns: name, order number and request. Here's what it looks like: name | order id | request ---------|----------|---------------- Jonah | 3310 | refund Priya | 5127 | change address Say you ask the assistant to fill in a row for Amara, and they send back a sentence. "The customer is Amara, and she's asking where order 4821 is." An LLM does the same thing, ask it for Amara's details and it replies in a sentence but the Kora Home app fills in the support log automatically, so it needs the name, the order number and the request as three separate values. In this section, we’ll look at three approaches that can help us achieve that. Approach One: Asking in Plain Words The simplest approach is to ask. “Pull out the customer’s name, the order number and what they want.” The model replies with something like this. Sure The customer is Amara, she’s asking about order 4821, and she wants to know where it is. The answer is correct, but it’s still a sentence. And the next call might word it differently, “Amara wants an update on 4821.” The app would have to find the name and order number inside a different sentence every time because the shape the LLM response might come back in is not fixed anywhere. So this approach fails. Approach Two: Asking for JSON The next approach is to ask the model to reply in a format that comes with labels built in. The most common one is JSON. JSON is a text format for data, where each piece of information is written next to a label. Remember the support log table, here’s Amara’s record as JSON, the way we'd like the model to reply. {"name": "Amara", "order id": "4821", "request": "order status"} Each piece of information is paired with a label, and each label matches a column heading in the support log. In JSON, the labels are called keys , and the information paired with each key is its value . So order id is a key, and 4821 is its value. The model’s reply arrives as plain text, though. Before the app can pick out each value, it has to turn that text into something Python can work with. Parsing is turning text into data your program can use. The piece of code that does it is called a parser . In Python, the parser is json.loads . Here it is working on the clean reply. reply = '{"name": "Amara", "order id": "4821", "request": "order status"}' record = json.loads reply print record "name" Amara print record "order id" 4821 print record "request" order status After parsing, record is a Python dictionary , a structure where you look up each value by its key. Asking for record "order id" gives back 4821 . The app does this once for each column, and Amara’s row gets added to the support log. name | order id | request ---------|----------|---------------- Jonah | 3310 | refund Priya | 5127 | change address Amara | 4821 | order status So the plan is to ask for JSON and parse whatever comes back. Most of the time, the reply looks like the clean one above. But every so often, it looks like this. Clean reply {"name": "Amara", "order id": "4821", "request": "order status"} Bad reply Sure Here's the record: {"name": "Amara", "order": "4821", "request": "order status"} All the information is there in both. Put them side by side, though, and two differences show up. - The first is the sentence at the top of the bad reply. The parser reads from the very first character, and it expects that character to be a curly brace. It finds the “S” of “Sure ” and stops. Here’s what Python prints. json.decoder.JSONDecodeError: Expecting value: line 1 column 1 char 0 “Line 1 column 1” is Python pointing at the very first character, the “S”. - The second is the middle key. The clean reply says order id . The bad reply says order . Say the sentence at the top were removed and the parse went through. The app would still ask for record "order id" , like it does for every row, and this record holds a key called order . Here's what Python prints then. KeyError: 'order id' A KeyError means the app asked for a key the record lacks. Either error stops the app, and Amara's row stays empty in the support log. So why would the model send a bad reply at all, when we asked for JSON? The answer goes back to the prediction machine. The model writes one token at a time, and at each step it picks from a range of possibilities, ranked by how likely each one is. Here’s roughly what that range could look like for the very first token of the reply. The numbers are made up to show the idea. First token Likelihood { 62% Sure 21% Here 11% Okay 6% Asking for JSON pushed { to the top of the list. But “Sure” is still on it, because the model has seen countless helpful replies that start that way. Most calls pick { . Now and then, a call picks “Sure”, and that’s the bad reply. You might be thinking of temperature here. At temperature 0, the model always picks the top option. But every customer message is different, and each one reshuffles the list. For some messages, “Sure” lands at the top, and temperature 0 then picks it every time for that message. So Approach Two works most of the time. In an app handling thousands of messages a day, “most of the time” still means dozens of broken rows. Both approaches so far have one thing in common. We ask, and the model decides what to write. Approach Three takes that decision about shape away from the model entirely. Approach Three: Handing the Model a Form Go back to the phone call. So far, the person on the other end can answer however they like. Structured outputs changes that. Before they answer, you hand them a form with boxes labelled name, order ID, request. They can only write inside the boxes. The form is called a schema. A schema is a written description of the shape you expect an answer to have. It lists each field by name, says what type of value goes in it, and says which fields must be filled in. Here’s the schema for Amara’s record, written in Python. record schema = { "type": "object", "properties": { "name": {"type": "string"}, "order id": {"type": "string"}, "request": {"type": "string"}, }, "required": "name", "order id", "request" , "additionalProperties": False, } That’s a lot but let’s go line by line like we always do: - "type": "object" means the reply is one JSON object, one pair of curly braces. - properties lists the boxes on the form, name , order id and request . Each one is a string , which means text. The order ID is text too, since order numbers are labels and some shops put letters in them. - required lists the boxes that must be filled in. Here, all three. - "additionalProperties": False means the reply holds these three boxes and only these three. How the Form Is Enforced When you send a schema with your request, the API checks every option on the model's list against the form, and it removes the ones that would break it. Here's the same first-token list from Approach Two, with a schema attached. First token With the schema { allowed Sure removed Here removed Okay removed With “Sure” removed, the model can only pick { . The same thing happens at every step after that. Here’s the step where the bad reply wrote order . Next token With the schema "order id" allowed "order" removed "orderNumber" removed The schema says the second key is order id , so that’s the only key the model can write. Inside the quote marks for each value, the model chooses freely. That’s where Amara , 4821 and order status come from. So at every step, two things happen. The model ranks its options like it always does. Then the API removes every option that would break the form, and the model picks from whatever is left. The model still does the writing. The schema decides what it's allowed to write. That process is constrained decoding. Constrained decoding is how the API limits the model's choice of next token, at every step, to the tokens that fit your schema. The name comes from two words. Decoding is the model picking its tokens one at a time to build a reply. Constrained means the choices are limited. Put together, the model picks its tokens from a limited list. Sending the Request The schema goes into the call through a parameter called output config . The rest is the same call from aie 2.0, with model , max tokens and a messages array. We’ll work through a dedicated build with this so you can implement it yourself. For now, just the gist of it. python import json import anthropic client = anthropic.Anthropic amara message = "Hi, this is Amara. My order 4821 was due last week and I'm still waiting. Where is it?" response = client.messages.create model="claude-haiku-4-5", max tokens=1024, messages= {"role": "user", "content": f"Turn this customer message into a support record: {amara message}"} , output config={ "format": { "type": "json schema", "schema": record schema, } }, Reading the Reply Here’s what comes back in the first content block, consistently. {"name": "Amara", "order id": "4821", "request": "order status"} It matches the clean reply from Approach Two exactly. So the app parses it the same way. record = json.loads response.content 0 .text print record "name" Amara print record "order id" 4821 print record "request" order status And Amara's row goes into the support log. name | order id | request ---------|----------|---------------- Jonah | 3310 | refund Priya | 5127 | change address Amara | 4821 | order status The Schema’s Limit The schema solves the problem we started with. Our replies now come back in the right shape, and Amara’s row goes into the support log. But the schema only controls one thing, which tokens the model is allowed to pick. There are two situations where we can still have issues with our reply, even with the schema in place. We’ll look at both here: Reply that gets cut off: In aie 2.0, we saw that max tokens sets a ceiling on how long a reply can be, and a reply that runs longer gets cut off, even mid-sentence. The schema has zero say over that ceiling. So say max tokens had been set far too low for this call, the reply would come back like this. {"name": "Amara", "order id": "48 The reply stops partway through the order number. When the app tries to parse it, Python prints this. json.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 31 char 30 “Unterminated string” means Python found an opening quote mark with no closing one. So how would the app know a reply was cut off, before trying to parse it? The answer is in the response object. In aie 2.0, we looked at two of its parts, content for the reply and usage for the token counts. It has one more part we skipped, and this is where it matters. The stop reason is a field in the response object that tells you why the model stopped writing. You read it with response.stop reason . Here’s how to read it. print response.stop reason For the full reply, this prints end turn , which means the model finished its answer. For the cut-off reply, it prints max tokens , which means the model hit the ceiling. There’s a third value to know about, refusal , which means the model declined the request for safety reasons. A refused reply may follow a different shape too. So before parsing, the app checks that the model finished. if response.stop reason == "end turn": record = json.loads response.content 0 .text The if line lets the parse happen only when the stop reason is end turn . A cut-off or refused reply skips the parse, and the app can handle it separately, for example by trying the call again with a higher max tokens . Reply with the right shape and a wrong value: The schema controls the shape, and the model still writes the values. Say Amara’s message had mentioned two orders. Hi, this is Amara. My order 4821 was due last week and I’m still waiting. I also returned order 4790. Where is my order? The model might reply with this. {"name": "Amara", "order id": "4790", "request": "order status"} The shape is perfect, it parses cleanly and goes straight into the support log. name | order id | request ---------|----------|---------------- Jonah | 3310 | refund Priya | 5127 | change address Amara | 4790 | order status But 4790 is the order she returned, and 4821 is the one she’s asking about. The row looks right and holds the wrong order. Checking whether values are correct is the subject of aie 4.0. This is important because as an AI Engineer, any time a model's answer feeds into code, whether it's saved to a database, shown on a screen or passed to another step, you need it in a fixed shape. A schema is how you get it, and the stop reason is how you confirm the reply finished before you trust that shape. Asking in plain words gave us a sentence, asking for JSON gave us clean replies most of the time, and occasional bad ones that broke the app. Handing the model a form, a schema, gave us the right shape every time, because the API removes every token that would break the form. The stop reason tells us whether the reply finished and the values inside still need checking, which is a job for later in the series. Back at Kora Home, Amara's message is now a clean row in the support log, and the app knows she wants to know where order 4821 is. But the answer is in the shop's database, outside anything the model can see, and once the app does have an answer, Amara is still looking at an empty chat window, waiting. Those are jobs two and three. In the next article, we cover function calling, which lets the model ask your app to look up order 4821, and streaming, which gets the answer onto Amara's screen while it's still being written.