{"slug": "aie-2-2-structured-outputs-getting-reliable-json-from-an-llm", "title": "aie_2.2: structured outputs, getting reliable json from an llm", "summary": "A developer's AI engineering series (aie_2.2) explains how to get reliable JSON from an LLM using structured outputs, framing the model as a new assistant that must return data in a consistent layout for the rest of an application to consume. The piece contrasts structured outputs with function calling and streaming, which are slated for the next installment, and recaps foundational concepts including tokens, statelessness, and context.", "body_md": "Before I introduce anything, I want to take you back to where we left off.\n\nIn aie_2.0, we went inside an LLM API call. We looked at what you send (the messages array and a handful of parameters) and what you get back (the response object). In aie_2.1, we took those pieces and built what we called a conversation engine, a small program you can chat with in your terminal. If you followed along with the build, then you’ve had a back-and-forth conversation with an LLM, running on your code.\n\nWhat we built there is a chatbot. And a chatbot is the simplest thing you can build with an LLM. You type something, the model writes something back and you read it.\n\nMost applications ask much more of a model than that. In a real app, one part among many. It gets handed a task and whatever it produces gets passed along to other parts of the app, *to a database, to a screen, to a customer*. So its answers have to fit into work that's already happening inside of said application.\n\nThat is a bit abstract, so I’ll solidify it. Think about what it's like to bring a new assistant into a small business. They're smart, they write well, and they're eager to help. By lunchtime, you've handed them three tasks. Each task comes with something you'd expect from them.\n\n*If you ask them to fill in a spreadsheet, what do you need from them?* The right information, in the right columns, laid out the same way every  time.\n\n*If you ask them about something only you have the records for, like a customer’s order, what should they do?* Go and look it up.\n\n*And if the answer takes a while, what would you like while you wait?* Some sign that they’re working on it, so you know something is happening.\n\nThe model in your app is that new assistant. It's smart, it writes well, and the app hands it tasks. So the app needs the same three things from the model that you'd need from your assistant.\n\nIt needs answers laid out the same way every time, so the rest of the app can use them. It needs a way for the model to ask for information that only the app's records hold. And when an answer takes a while, it needs to show the person waiting that something is happening.\n\nThe LLM API has a feature for each one. **Structured outputs** gets answers in the same layout every time. **Function calling** lets the model ask your app to look things up. **Streaming** shows the answer while it's being written.\n\nThis article is all about the first one, structured outputs. Function calling and streaming get the next article.\n\nBefore we get into it, let’s refresh a few terms. Every one of these comes back in this article, so I’d like them fresh in your mind before we build on them.\n\n## A Quick Refresher\n\nIn aie_1.0, we learned that an LLM is a ***prediction machine***.\n\n*A **Large Language Model (LLM)** is a prediction machine that is trained on a large amount of text and is capable of generating coherent, contextually appropriate language by predicting likely continuations of any input given.*\n\nThe key word is still *predict*. The model writes its answer one small piece at a time. At every step, it looks at everything it has so far and picks the most likely next piece. Each of those small pieces is called a ***token***.\n\n*A **token** is the smallest unit the model works with. Think of it as a chunk of text. “Hello” is one token. “Unbelievable” might be three.*\n\nWe also learned that LLMs have amnesia. The model forgets each call the moment it ends, and every new call arrives as if it’s meeting you for the first time.\n\n*A system is **stateless** when each request (message) is handled completely on its own, with zero memory of any request that came before it.*\n\nSo to make the model *\"remember\"*, the app sends the whole conversation back on every call. That conversation is part of the model's context, everything the model can see at the moment it writes a response.\n\nIn aie_2.0, we pictured all of this as a phone call where the person on the other end has amnesia. Every time you call, they pick up with zero idea who you are. The only way to have a conversation is to read them the full transcript from the beginning, every single time.\n\nThat transcript is the messages array.\n\n*The **messages array** is the list of all that you send to the model on a single API call. It is structured as a sequence of messages, each one labelled with who sent it.*\n\nEach label is a role. `user` is whoever is on the other end of your product, like your customer. `assistant` is the model. Alongside the messages array, you send parameters.\n\n**Parameters** are settings you include with every API call that shape how the model responds.\n\nWe covered three of them. `model` picks which version of Claude to use. `temperature` controls how tightly the model sticks to the most likely next word. At 0, it always picks the most probable one. And `max_tokens` sets a hard ceiling on how long the reply can be, measured in tokens.\n\nFinally, what comes back is the response object. We described it as a delivery with a receipt attached. The package is the model’s reply, and the receipt tells you what it cost.\n\n- `response.content` is a list of content blocks, where the model’s reply lives.`response.content[0].text` takes the first block and pulls out the words.\n- `response.usage` holds the token counts,`input_tokens` for what you sent and`output_tokens` for what the model wrote back.\n\nThe response object has one more field we'll need in this article. We'll meet it when it comes up. Also, keep that phone call in mind as we go. Structured outputs changes the call in one small way, and we'll come back to it.\n\n## From a Chatbot to a Real App\n\nTo see what a real app asks of an LLM, let’s follow one example from the start of this article to the end.\n\nKora Home is a small online shop that sells home goods, things like lamps, throws and ceramic mugs. They want an assistant that answers customer messages. One morning, a customer named Amara sends this.\n\n**Amara:** Hi, this is Amara. My order 4821 was due last week and I’m still waiting. Where is it?\n\nFor the assistant to handle Amara’s message properly, the app has three jobs to do.\n\n- **Job one.** ***Turn her message into a record.*** Kora Home keeps a support log in a database table. Each row holds three things: the customer's name, the order number and what they want. So the app needs three separate, clean values out of Amara's message.`Amara` ,`4821` and`order status` .\n- **Job two.** ***Find out where order 4821 is.*** That information is stored in Kora Home’s database, on Kora Home’s computers.\n- **Job three.** ***Show Amara the answer.*** The reply appears in a chat window on her screen.\n\nWith only what we learned in aie_2.0, each job has its issue:\n\n- For job one, ***the model replies in free text.*** The wording changes from call to call, and a program needs the same values in the same places every time.\n- For job two, ***the model can only work with what’s in its context.*** Order 4821 is stored in the shop’s database, outside the model’s context.\n- For job three, ***a long answer takes several seconds to write*** , and Amara looks at an empty chat window the whole time.\n\nHere’s how the three features map onto the three jobs.\n\n- **Structured outputs** handles job one.\n- **Function calling** handles job two.\n- **Streaming** handles job three.\n\nIn this lesson, we take job one. Jobs two and three are the next article, and Amara's message is what we’ll use in those.\n\n*This is important because as an AI Engineer, most of your work happens in the space between the model and the rest of your app. The model writes text and your job is getting that text into a shape your code can use, connecting the model to the information it needs, and delivering its answers to people in a way that makes sense to use. These three features are your first tools for that work.*\n\n## Structured Outputs\n\nLet’s look at turning Amara’s message into a record. Remember the assistant with the spreadsheet, Kora Home's support log is a table with three columns: name, order number and request. Here's what it looks like:\n\n```\nname     | order_id | request\n---------|----------|----------------\nJonah    | 3310     | refund\nPriya    | 5127     | change address\n```\n\nSay you ask the assistant to fill in a row for Amara, and they send back a sentence. *\"The customer is Amara, and she's asking where order 4821 is.\"*\n\nAn LLM does the same thing, ask it for Amara's details and it replies in a sentence but the Kora Home app fills in the support log automatically, so it needs the name, the order number and the request as three separate values.\n\nIn this section, we’ll look at three approaches that can help us achieve that.\n\n### Approach One: Asking in Plain Words\n\nThe simplest approach is to ask. *“Pull out the customer’s name, the order number and what they want.”* The model replies with something like this.\n\n*Sure! The customer is Amara, she’s asking about order 4821, and she wants to know where it is.*\n\nThe answer is correct, but it’s still a sentence. And the next call might word it differently, *“Amara wants an update on 4821.”* The app would have to find the name and order number inside a different sentence every time because the shape the LLM response might come back in is not fixed anywhere. So this approach fails.\n\n### Approach Two: Asking for JSON\n\nThe next approach is to ask the model to reply in a format that comes with labels built in. The most common one is JSON.\n\n**JSON** is a text format for data, where each piece of information is written next to a label.\n\nRemember the support log table, here’s Amara’s record as JSON, the way we'd like the model to reply.\n\n```\n{\"name\": \"Amara\", \"order_id\": \"4821\", \"request\": \"order status\"}\n```\n\nEach piece of information is paired with a label, and each label matches a column heading in the support log. In JSON, the labels are called **keys**, and the information paired with each key is its **value**. So `order_id` is a key, and `4821` is its value.\n\nThe model’s reply arrives as plain text, though. Before the app can pick out each value, it has to turn that text into something Python can work with.\n\n**Parsing** is turning text into data your program can use. The piece of code that does it is called a **parser**.\n\nIn Python, the parser is `json.loads`. Here it is working on the clean reply.\n\n```\nreply = '{\"name\": \"Amara\", \"order_id\": \"4821\", \"request\": \"order status\"}'\n\nrecord = json.loads(reply)\n\nprint(record[\"name\"])      # Amara\nprint(record[\"order_id\"])  # 4821\nprint(record[\"request\"])   # order status\n```\n\nAfter parsing, `record` is a Python **dictionary**, a structure where you look up each value by its key. Asking for `record[\"order_id\"]` gives back `4821`. The app does this once for each column, and Amara’s row gets added to the support log.\n\n```\nname     | order_id | request\n---------|----------|----------------\nJonah    | 3310     | refund\nPriya    | 5127     | change address\nAmara    | 4821     | order status\n```\n\nSo the plan is to ask for JSON and parse whatever comes back. Most of the time, the reply looks like the clean one above. But every so often, it looks like this.\n\n**Clean reply**\n\n```\n{\"name\": \"Amara\", \"order_id\": \"4821\", \"request\": \"order status\"}\n```\n\n**Bad reply**\n\n```\nSure! Here's the record:\n{\"name\": \"Amara\", \"order\": \"4821\", \"request\": \"order status\"}\n```\n\nAll the information is there in both. Put them side by side, though, and two differences show up.\n\n- The first is the sentence at the top of the bad reply. The parser reads from the very first character, and it expects that character to be a curly brace. It finds the “S” of “Sure!” and stops. Here’s what Python prints. \n\n```\njson.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)\n```\n\n “Line 1 column 1” is Python pointing at the very first character, the “S”.\n- The second is the middle key. The clean reply says `order_id` . The bad reply says`order` . Say the sentence at the top were removed and the parse went through. The app would still ask for`record[\"order_id\"]` , like it does for every row, and this record holds a key called`order` . Here's what Python prints then.\n\n```\nKeyError: 'order_id'\n```\n\n A `KeyError` means the app asked for a key the record lacks. Either error stops the app, and Amara's row stays empty in the support log.\n\n*So why would the model send a bad reply at all, when we asked for JSON?* The answer goes back to the prediction machine. The model writes one token at a time, and at each step it picks from a range of possibilities, ranked by how likely each one is. Here’s roughly what that range could look like for the very first token of the reply. The numbers are made up to show the idea.\n\n```\nFirst token     Likelihood\n{               62%\nSure            21%\nHere            11%\nOkay             6%\n```\n\nAsking for JSON pushed `{` to the top of the list. But “Sure” is still on it, because the model has seen countless helpful replies that start that way. Most calls pick `{`. Now and then, a call picks “Sure”, and that’s the bad reply.\n\n*You might be thinking of temperature here.* At temperature 0, the model always picks the top option. But every customer message is different, and each one reshuffles the list. For some messages, “Sure” lands at the top, and temperature 0 then picks it every time for that message.\n\nSo Approach Two works most of the time. *In an app handling thousands of messages a day, “most of the time” still means dozens of broken rows.* Both approaches so far have one thing in common. We ask, and the model decides what to write. Approach Three takes that decision about shape away from the model entirely.\n\n### Approach Three: Handing the Model a Form\n\nGo back to the phone call. So far, the person on the other end can answer however they like. *Structured outputs changes that.* Before they answer, you hand them a form with boxes labelled *name, order ID, request.* They can only write inside the boxes.\n\nThe form is called a **schema.**\n\n*A **schema** is a written description of the shape you expect an answer to have. It lists each field by name, says what type of value goes in it, and says which fields must be filled in.*\n\nHere’s the schema for Amara’s record, written in Python.\n\n```\nrecord_schema = {\n    \"type\": \"object\",\n    \"properties\": {\n        \"name\": {\"type\": \"string\"},\n        \"order_id\": {\"type\": \"string\"},\n        \"request\": {\"type\": \"string\"},\n    },\n    \"required\": [\"name\", \"order_id\", \"request\"],\n    \"additionalProperties\": False,\n}\n```\n\nThat’s a lot but let’s go line by line like we always do:\n\n- `\"type\": \"object\"` means the reply is one JSON object, one pair of curly braces.\n- `properties` lists the boxes on the form,`name` ,`order_id` and`request` . Each one is a`string` , which means text. The order ID is text too, since order numbers are labels and some shops put letters in them.\n- `required` lists the boxes that must be filled in. Here, all three.\n- `\"additionalProperties\": False` means the reply holds these three boxes and only these three.\n\n#### How the Form Is Enforced\n\nWhen you send a schema with your request, the API checks every option on the model's list against the form, and it removes the ones that would break it. Here's the same first-token list from Approach Two, with a schema attached.\n\n```\nFirst token     With the schema\n{               allowed\nSure            removed\nHere            removed\nOkay            removed\n```\n\nWith “Sure” removed, the model can only pick `{`. The same thing happens at every step after that. Here’s the step where the bad reply wrote `order`.\n\n```\nNext token      With the schema\n\"order_id\"      allowed\n\"order\"         removed\n\"orderNumber\"   removed\n```\n\nThe schema says the second key is `order_id`, so that’s the only key the model can write. Inside the quote marks for each value, the model chooses freely. That’s where `Amara`, `4821` and `order status` come from.\n\nSo at every step, two things happen. The model ranks its options like it always does. Then the API removes every option that would break the form, and the model picks from whatever is left. The model still does the writing. The schema decides what it's allowed to write.\n\nThat process is **constrained decoding.**\n\n**Constrained decoding** is how the API limits the model's choice of next token, at every step, to the tokens that fit your schema.\n\nThe name comes from two words. ***Decoding*** is the model picking its tokens one at a time to build a reply. *** Constrained*** means the choices are limited. Put together, the model picks its tokens from a limited list.\n\n#### Sending the Request\n\nThe schema goes into the call through a parameter called `output_config`. The rest is the same call from aie_2.0, with `model`, `max_tokens` and a messages array. We’ll work through a dedicated build with this so you can implement it yourself. For now, just the gist of it.\n\n``` python\nimport json\nimport anthropic\n\nclient = anthropic.Anthropic()\n\namara_message = \"Hi, this is Amara. My order 4821 was due last week and I'm still waiting. Where is it?\"\n\nresponse = client.messages.create(\n    model=\"claude-haiku-4-5\",\n    max_tokens=1024,\n    messages=[\n        {\"role\": \"user\", \"content\": f\"Turn this customer message into a support record: {amara_message}\"}\n    ],\n    output_config={\n        \"format\": {\n            \"type\": \"json_schema\",\n            \"schema\": record_schema,\n        }\n    },\n)\n```\n\n#### Reading the Reply\n\nHere’s what comes back in the first content block, consistently.\n\n```\n{\"name\": \"Amara\", \"order_id\": \"4821\", \"request\": \"order status\"}\n```\n\nIt matches the clean reply from Approach Two exactly. So the app parses it the same way.\n\n```\nrecord = json.loads(response.content[0].text)\n\nprint(record[\"name\"])      # Amara\nprint(record[\"order_id\"])  # 4821\nprint(record[\"request\"])   # order status\n```\n\nAnd Amara's row goes into the support log.\n\n```\nname     | order_id | request\n---------|----------|----------------\nJonah    | 3310     | refund\nPriya    | 5127     | change address\nAmara    | 4821     | order status\n```\n\n## The Schema’s Limit\n\nThe schema solves the problem we started with. Our replies now come back in the right shape, and Amara’s row goes into the support log. But the schema only controls one thing, which tokens the model is allowed to pick. There are two situations where we can still have issues with our reply, even with the schema in place. We’ll look at both here:\n\n### Reply that gets cut off:\n\nIn aie_2.0, we saw that `max_tokens` sets a ceiling on how long a reply can be, and a reply that runs longer gets cut off, even mid-sentence. The schema has zero say over that ceiling. So say `max_tokens` had been set far too low for this call, the reply would come back like this.\n\n```\n{\"name\": \"Amara\", \"order_id\": \"48\n```\n\nThe reply stops partway through the order number. When the app tries to parse it, Python prints this.\n\n```\njson.decoder.JSONDecodeError: Unterminated string starting at: line 1 column 31 (char 30)\n```\n\n“Unterminated string” means Python found an opening quote mark with no closing one.\n\n*So how would the app know a reply was cut off, before trying to parse it?* The answer is in the response object. In aie_2.0, we looked at two of its parts, `content` for the reply and `usage` for the token counts. It has one more part we skipped, and this is where it matters.\n\n*The **stop reason** is a field in the response object that tells you why the model stopped writing. You read it with* `response.stop_reason`*.*\n\nHere’s how to read it.\n\n```\nprint(response.stop_reason)\n```\n\nFor the full reply, this prints `end_turn`, which means the model finished its answer. For the cut-off reply, it prints `max_tokens`, which means the model hit the ceiling. There’s a third value to know about, `refusal`, which means the model declined the request for safety reasons. A refused reply may follow a different shape too.\n\nSo before parsing, the app checks that the model finished.\n\n```\nif response.stop_reason == \"end_turn\":\n    record = json.loads(response.content[0].text)\n```\n\nThe `if` line lets the parse happen only when the stop reason is `end_turn`. A cut-off or refused reply skips the parse, and the app can handle it separately, for example by trying the call again with a higher `max_tokens`.\n\n### Reply with the right shape and a wrong value:\n\nThe schema controls the shape, and the model still writes the values. Say Amara’s message had mentioned two orders.\n\n*Hi, this is Amara. My order 4821 was due last week and I’m still waiting. I also returned order 4790. Where is my order?*\n\nThe model might reply with this.\n\n```\n{\"name\": \"Amara\", \"order_id\": \"4790\", \"request\": \"order status\"}\n```\n\nThe shape is perfect, it parses cleanly and goes straight into the support log.\n\n```\nname     | order_id | request\n---------|----------|----------------\nJonah    | 3310     | refund\nPriya    | 5127     | change address\nAmara    | 4790     | order status\n```\n\nBut `4790` is the order she returned, and `4821` is the one she’s asking about. The row looks right and holds the wrong order. Checking whether values are correct is the subject of aie_4.0.\n\n*This is important because as an AI Engineer, any time a model's answer feeds into code, whether it's saved to a database, shown on a screen or passed to another step, you need it in a fixed shape. A schema is how you get it, and the stop reason is how you confirm the reply finished before you trust that shape.*\n\nAsking in plain words gave us a sentence, asking for JSON gave us clean replies most of the time, and occasional bad ones that broke the app. Handing the model a form, a schema, gave us the right shape every time, because the API removes every token that would break the form. The stop reason tells us whether the reply finished and the values inside still need checking, which is a job for later in the series.\n\nBack at Kora Home, Amara's message is now a clean row in the support log, and the app knows she wants to know where order 4821 is. But the answer is in the shop's database, outside anything the model can see, and once the app does have an answer, Amara is still looking at an empty chat window, waiting.\n\nThose are jobs two and three. In the next article, we cover function calling, which lets the model ask your app to look up order 4821, and streaming, which gets the answer onto Amara's screen while it's still being written.", "url": "https://wpnews.pro/news/aie-2-2-structured-outputs-getting-reliable-json-from-an-llm", "canonical_source": "https://heymeraki.substack.com/p/aie_22-structured-outputs-getting", "published_at": "2026-09-22 15:39:39+00:00", "updated_at": "2026-09-22 16:56:05.165786+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "developer-tools", "natural-language-processing"], "entities": [], "alternates": {"html": "https://wpnews.pro/news/aie-2-2-structured-outputs-getting-reliable-json-from-an-llm", "markdown": "https://wpnews.pro/news/aie-2-2-structured-outputs-getting-reliable-json-from-an-llm.md", "text": "https://wpnews.pro/news/aie-2-2-structured-outputs-getting-reliable-json-from-an-llm.txt", "jsonld": "https://wpnews.pro/news/aie-2-2-structured-outputs-getting-reliable-json-from-an-llm.jsonld"}}