{"slug": "getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained", "title": "Getting Reliable Structured Outputs from LLMs: Validation, Retries and Constrained Decoding", "summary": "A tutorial published on structured LLM output recommends a three-layer approach — a strict Pydantic contract, source-text validation, and bounded retries capped at three attempts — to catch errors that schema validation alone misses, such as a model returning 12,500 cents for an invoice that states EUR 1,250.00. The examples, run on Python 3.10.12 and Pydantic 2.14.1, define an Invoice model with strict mode and extra=\"forbid\", compare every validated object against the source invoice, and use constrained decoding through OpenAI's Responses API or vLLM with xgrammar to prevent shape failures from reaching validation.", "body_md": "In this article, you will learn how to get reliable structured outputs from large language models by validating them against a contract and the source text, retrying only when it helps, and constraining generation.\n\nTopics we will cover include:\n\n- How to define a strict Pydantic contract and check every validated object against the source it came from.\n- How to build a bounded retry loop that regenerates only on shape errors and keeps the SDK’s hidden transport retries under control.\n- How to use constrained decoding with OpenAI’s Responses API without dropping validation, and how to test each failure path with a mocked client.\n\nStructured output is a standard feature now. OpenAI’s Responses API takes a JSON schema or a Pydantic model and promises a response that fits it, vLLM does the same on a local server with xgrammar behind it, and the Python SDKs hand you a parsed object instead of a string. Most tutorials stop at the moment the JSON parses. That’s the wrong place to stop, and here’s the whole problem in four lines of invoice:\n\n```\nINVOICE 2026-0417\nBill to: Northwind Traders\nTotal due: EUR 1,250.00\nDue 30 days from issue.\n\n1234\n\nINVOICE 2026-0417Bill to: Northwind TradersTotal due: EUR 1,250.00Due 30 days from issue.\n```\n\nAsk a model for the invoice id, the currency, the total in cents and the line it took the total from, and you might get this back:\n\n```\n{\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": 12500, \"total_quote\": \"Total due: EUR 1,250.00\"}\n\n1\n\n{\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": 12500, \"total_quote\": \"Total due: EUR 1,250.00\"}\n```\n\nIt’s valid JSON, every key is present, the types are right, EUR is in the allowed set, and the quoted line really is in the invoice. The total is 12,500 cents, which is 125 euros, for an invoice that says 1,250. No parser catches that and no schema catches it, because 12500 is a perfectly good integer. The only thing that catches it is a comparison with the source text, and your application has to make that comparison itself.\n\nThe three terms at the end of the title are the three layers that do the catching. Validation means your code checks the returned object against a contract you own, then against the invoice it came from. Retries means a second attempt is allowed only when another attempt can fix what went wrong, with a hard stop at three. Constrained decoding means the generator can only emit tokens that keep the output inside your schema, so a whole class of shape failures never reaches validation. The examples below were run on Python 3.10.12 and Pydantic 2.14.1; the code extracts fields and nothing more, and never writes to a real system.\n\n## Define the Contract Before Calling the Model\n\nThe contract has four fields, and your application owns it, not the prompt:\n\n``` python\nfrom typing import Literal\nfrom pydantic import BaseModel, ConfigDict\n\nclass Invoice(BaseModel):\n    model_config = ConfigDict(strict=True, extra=\"forbid\")\n\n    invoice_id: str\n    currency: Literal[\"USD\", \"EUR\"] | None\n    total_cents: int | None\n    total_quote: str | None\n\n1234567891011\n\nfrom typing import Literalfrom pydantic import BaseModel, ConfigDict  class Invoice(BaseModel):    model_config = ConfigDict(strict=True, extra=\"forbid\")     invoice_id: str    currency: Literal[\"USD\", \"EUR\"] | None    total_cents: int | None    total_quote: str | None\n```\n\nEvery key is required, so a response that leaves one out fails. Three fields are nullable without a default, which is different from optional: the model may say “the invoice doesn’t state this” by sending null, but it may not stay silent and it may not invent a number to avoid a null. Two currencies is a tutorial scope, not a claim about invoices. Cents are stored as an integer so that 1,250.00 never becomes 1249.9999.\n\nThe two configuration options do most of the work. [Strict mode](https://docs.pydantic.dev/latest/concepts/strict_mode/) switches off Pydantic’s default coercion, which would otherwise accept the string `\"125000\"` for an integer field and convert it silently; in strict mode that string fails with the error type `int_type`, and so does the boolean `True`, which the default mode would turn into 1. `extra=\"forbid\"` turns an unexpected key into an `extra_forbidden` error instead of ignoring it. Feed `Invoice` four bad inputs and you get four distinct error types back:\n\n``` python\nfrom pydantic import ValidationError\n\nbad_inputs = [\n    {\"invoice_id\": \"2026-0417\", \"currency\": \"GBP\", \"total_cents\": 125000, \"total_quote\": \"x\"},\n    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": \"125000\", \"total_quote\": \"x\"},\n    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": True, \"total_quote\": \"x\"},\n    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": 125000, \"total_quote\": \"x\", \"vendor\": \"v\"},\n]\nfor item in bad_inputs:\n    try:\n        Invoice.model_validate(item)\n    except ValidationError as e:\n        print([(err[\"loc\"], err[\"type\"]) for err in e.errors()])\n\n12345678910111213\n\nfrom pydantic import ValidationError bad_inputs = [    {\"invoice_id\": \"2026-0417\", \"currency\": \"GBP\", \"total_cents\": 125000, \"total_quote\": \"x\"},    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": \"125000\", \"total_quote\": \"x\"},    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": True, \"total_quote\": \"x\"},    {\"invoice_id\": \"2026-0417\", \"currency\": \"EUR\", \"total_cents\": 125000, \"total_quote\": \"x\", \"vendor\": \"v\"},]for item in bad_inputs:    try:        Invoice.model_validate(item)    except ValidationError as e:        print([(err[\"loc\"], err[\"type\"]) for err in e.errors()])\n[(('currency',), 'literal_error')]\n[(('total_cents',), 'int_type')]\n[(('total_cents',), 'int_type')]\n[(('vendor',), 'extra_forbidden')]\n\n1234\n\n[(('currency',), 'literal_error')][(('total_cents',), 'int_type')][(('total_cents',), 'int_type')][(('vendor',), 'extra_forbidden')]\n```\n\nYou can use [dataclasses for application-owned data](https://machinelearningmastery.com/dataclasses-for-structured-application-data/), but model output needs validation before it becomes a trusted record, and `Invoice` is the gate it passes through. One thing the gate can’t hold: `Invoice.model_json_schema()` exports four properties, all required, `additionalProperties` false, and not a word about the invoice. The rule that the cents have to match the quoted line can’t be written in JSON Schema, so a decoder that enforces this schema perfectly will still produce the 12,500-cent total from the opening. A valid object is the entry ticket to the next check, not the result.\n\n## Validate the Object, Then Check It Against the Source\n\n`accept()` runs four steps and ends in one of four states: `accepted`, `needs_review`, `rejected` or `shape_error`.\n\n``` python\nimport json\nfrom dataclasses import dataclass\nfrom pydantic import ValidationError\n\n@dataclass\nclass Outcome:\n    status: str            # \"accepted\" | \"needs_review\" | \"rejected\" | \"shape_error\"\n    invoice: Invoice | None\n    reasons: list[str]\n\ndef accept(response_text: str, source_text: str, expected_invoice_id: str) -> Outcome:\n    # Step 1: parse and validate the shape. Any failure here is a shape error.\n    try:\n        candidate = Invoice.model_validate(json.loads(response_text))\n    except json.JSONDecodeError as e:\n        return Outcome(\"shape_error\", None, [f\"not JSON: {e.msg} at position {e.pos}\"])\n    except ValidationError as e:\n        reasons = [f\"{'.'.join(str(p) for p in err['loc'])}: {err['msg']}\" for err in e.errors()]\n        return Outcome(\"shape_error\", None, reasons)\n\n    reasons: list[str] = []\n\n    # Step 2: the quoted line has to exist in the source, verbatim.\n    if candidate.total_quote is not None and candidate.total_quote not in source_text:\n        reasons.append(\"total_quote does not appear in the source\")\n\n    # Step 3: identity and currency have to match what the source says.\n    if candidate.invoice_id != expected_invoice_id:\n        reasons.append(f\"invoice_id {candidate.invoice_id!r} is not {expected_invoice_id!r}\")\n    if candidate.currency is not None and candidate.currency not in source_text:\n        reasons.append(f\"currency {candidate.currency} is not stated in the source\")\n\n    # Step 4: the amount has to be consistent with the quoted line.\n    if candidate.total_cents is not None and candidate.total_quote is not None:\n        digits = \"\".join(ch for ch in candidate.total_quote if ch.isdigit())\n        if digits and int(digits) != candidate.total_cents:\n            reasons.append(f\"total_cents {candidate.total_cents} does not match the quoted line\")\n\n    if reasons:\n        return Outcome(\"rejected\", candidate, reasons)\n    if candidate.total_cents is None or candidate.currency is None:\n        return Outcome(\"needs_review\", candidate, [\"source did not state the amount or currency\"])\n    return Outcome(\"accepted\", candidate, [])\n\n123456789101112131415161718192021222324252627282930313233343536373839404142434445\n\nimport jsonfrom dataclasses import dataclassfrom pydantic import ValidationError  @dataclassclass Outcome:    status: str            # \"accepted\" | \"needs_review\" | \"rejected\" | \"shape_error\"    invoice: Invoice | None    reasons: list[str]  def accept(response_text: str, source_text: str, expected_invoice_id: str) -> Outcome:    # Step 1: parse and validate the shape. Any failure here is a shape error.    try:        candidate = Invoice.model_validate(json.loads(response_text))    except json.JSONDecodeError as e:        return Outcome(\"shape_error\", None, [f\"not JSON: {e.msg} at position {e.pos}\"])    except ValidationError as e:        reasons = [f\"{'.'.join(str(p) for p in err['loc'])}: {err['msg']}\" for err in e.errors()]        return Outcome(\"shape_error\", None, reasons)     reasons: list[str] = []     # Step 2: the quoted line has to exist in the source, verbatim.    if candidate.total_quote is not None and candidate.total_quote not in source_text:        reasons.append(\"total_quote does not appear in the source\")     # Step 3: identity and currency have to match what the source says.    if candidate.invoice_id != expected_invoice_id:        reasons.append(f\"invoice_id {candidate.invoice_id!r} is not {expected_invoice_id!r}\")    if candidate.currency is not None and candidate.currency not in source_text:        reasons.append(f\"currency {candidate.currency} is not stated in the source\")     # Step 4: the amount has to be consistent with the quoted line.    if candidate.total_cents is not None and candidate.total_quote is not None:        digits = \"\".join(ch for ch in candidate.total_quote if ch.isdigit())        if digits and int(digits) != candidate.total_cents:            reasons.append(f\"total_cents {candidate.total_cents} does not match the quoted line\")     if reasons:        return Outcome(\"rejected\", candidate, reasons)    if candidate.total_cents is None or candidate.currency is None:        return Outcome(\"needs_review\", candidate, [\"source did not state the amount or currency\"])    return Outcome(\"accepted\", candidate, [])\n```\n\nStep one is the parser and the contract, and it’s the only step that can return `shape_error`, which matters because a shape error is the only outcome the next section is allowed to retry. Step two checks that `total_quote` is in the invoice character for character. Step three checks the id you asked about and that the currency the model picked is actually printed in the text. Step four catches the opening’s mistake: the digits in “Total due: EUR 1,250.00” with the separators stripped are 125000; the model sent 12500, so the object is `rejected` with the reason “total_cents 12500 does not match the quoted line”. For a real invoice format you’d write a check that understands your decimals; the principle holds either way: the number has to be derivable from the evidence or it isn’t accepted.\n\n`needs_review` is where a null goes when the invoice genuinely didn’t state an amount or a currency; under this policy a missing fact goes to a person, not to a guessed zero. `rejected` is terminal, because another call to the model can’t change what the invoice says. Expected values in the test fixtures are written by hand from the source text, never copied from a model’s output.\n\nOne boundary before the loop. Everything here starts from text that already exists. When you’re [extracting text from scanned invoices](https://apryse.com/blog/ocr-in-python), inspect the transcription before you ask the model to populate the schema, because a wrong digit upstream passes every check downstream as long as the object is internally consistent. What happens to a `shape_error` from `accept()` is the next question, and the answer is a policy rather than a loop that runs until something parses.\n\n## Retry Only When Another Attempt Can Help\n\n`extract_with_retries()` has three exits, and only one of them is a retry. A `shape_error` gets another attempt with the errors fed back. `accepted`, `needs_review` and `rejected` are terminal, because the source won’t change. A refusal or a configuration error stops at once, because repeating the request won’t help.\n\n``` python\nfrom typing import Callable\n\nMAX_ATTEMPTS = 3  # the first attempt plus at most two regenerations\n\nclass GenerationRefused(Exception):\n    \"\"\"The provider refused or the request is not repairable by another attempt.\"\"\"\n\ndef extract_with_retries(generate: Callable[[str, dict, list[str]], str],\n                         source_text: str, expected_invoice_id: str) -> tuple[Outcome, int]:\n    schema = Invoice.model_json_schema()\n    feedback: list[str] = []\n    attempts = 0\n    while attempts < MAX_ATTEMPTS:\n        attempts += 1\n        try:\n            response_text = generate(source_text, schema, feedback)\n        except GenerationRefused as e:\n            return Outcome(\"rejected\", None, [f\"stopped: {e}\"]), attempts\n        outcome = accept(response_text, source_text, expected_invoice_id)\n        if outcome.status != \"shape_error\":\n            return outcome, attempts      # terminal: the source will not change\n        feedback = outcome.reasons[:5]    # bounded: the relevant errors, nothing else\n    return Outcome(\"rejected\", None, [\"attempt budget exhausted\"]), attempts\n\n12345678910111213141516171819202122232425\n\nfrom typing import Callable MAX_ATTEMPTS = 3  # the first attempt plus at most two regenerations  class GenerationRefused(Exception):    \"\"\"The provider refused or the request is not repairable by another attempt.\"\"\"  def extract_with_retries(generate: Callable[[str, dict, list[str]], str],                         source_text: str, expected_invoice_id: str) -> tuple[Outcome, int]:    schema = Invoice.model_json_schema()    feedback: list[str] = []    attempts = 0    while attempts < MAX_ATTEMPTS:        attempts += 1        try:            response_text = generate(source_text, schema, feedback)        except GenerationRefused as e:            return Outcome(\"rejected\", None, [f\"stopped: {e}\"]), attempts        outcome = accept(response_text, source_text, expected_invoice_id)        if outcome.status != \"shape_error\":            return outcome, attempts      # terminal: the source will not change        feedback = outcome.reasons[:5]    # bounded: the relevant errors, nothing else    return Outcome(\"rejected\", None, [\"attempt budget exhausted\"]), attempts\n```\n\n`generate` is whatever calls your model; it gets the invoice, the schema and the feedback list, and returns text. The schema and the source never change between attempts. The feedback is capped at five reasons, enough to tell the model that `vendor` isn’t a field or that `total_cents` has to be an integer, and not enough to turn the prompt into a transcript of everything that went wrong. Use the [field-level validation errors](https://machinelearningmastery.com/the-complete-guide-to-pydantic-for-python-developers/) to identify the failed check and send only that; each error carries a `loc`, a `type` and a `msg`, and the message is usually all the model needs. Every candidate is validated from scratch, because a repair call can change the amount while it fixes the key.\n\nThe budget of three is only real if you also own the transport retries. The OpenAI Python SDK retries connection errors, timeouts, 409, 429 and 5xx responses twice by default with exponential backoff, so three application attempts can be nine HTTP requests without your code noticing. Set `max_retries=0` on the client for a budget you can demonstrate, add a per-call timeout (the default is ten minutes, far too long for an extraction) and keep an overall deadline in the loop; the [SDK retry configuration](https://github.com/openai/openai-python#retries) is documented per client and per request. Three attempts is this tutorial’s policy, not a universal optimum; what matters is counting the hidden work and stopping on the terminal cases, which the table below sorts.\n\n**Table 1. Match the failure to the next step.**\n\n| Failure | What to check | Next step in this tutorial | \n|---|---|---|\n| Malformed complete response, or shape or type mismatch | The parser or validator error and the remaining application budget | Bounded content regeneration with the source, the contract and the relevant errors; validate again | \n| Valid shape, wrong or unsupported source value | The source text and the acceptance rule | `rejected` , or a specifically justified bounded re-extraction; never accept because it parses | \n| Missing or ambiguous source fact sent as null | Whether the business information exists in the source | `needs_review` ; don’t invent a default | \n| Refusal or policy-filtered generation | The provider’s documented outcome | Stop or escalate; don’t rephrase to defeat the refusal | \n| Truncation or unfinished stream | The completion status and its reason | Don’t commit; adjust a justified output budget or simplify scope within the deadline | \n| Unsupported schema or model, invalid credentials, other configuration error | The exact API error | Fix the configuration; no blind content-repair loop | \n| Eligible transient transport failure | The SDK’s current retry behavior, backoff and budget | A controlled transport retry, kept distinct from content repair | \n| Application budget exhausted | Total attempts and elapsed deadline | Explicit failure or review; no extra call and no downstream write | \n\nA loop like this fixes shape failures. It can’t fix a model that was never constrained, and that’s the third layer.\n\n## Constrain Generation Without Dropping Validation\n\nAsking for JSON in the prompt and restricting what the generator can emit are different things. A prompt is a request, and the model can still answer with a code fence, a stray sentence or an extra key. A schema constraint runs inside decoding: at each step the generator may only choose from the tokens that keep the output valid. Adrian Tam’s walkthrough of [decoding strategies](https://machinelearningmastery.com/decoding-strategies-and-output-control/) puts it in one line: “The idea is to mask out tokens that would make the output invalid”; this example uses a supported schema constraint rather than implementing the masking. On that path the malformed, extra-key and wrong-type cases from the fixtures stop occurring, because the decoder can’t produce them.\n\n```\n# Documentation-derived; not executed against a paid API for this tutorial.\ndef extract_constrained(client, model: str, source_text: str, expected_invoice_id: str) -> Outcome:\n    response = client.responses.parse(\n        model=model,\n        input=[\n            {\"role\": \"system\", \"content\": \"Extract the invoice fields. Use null when the text does not state a value.\"},\n            {\"role\": \"user\", \"content\": source_text},\n        ],\n        text_format=Invoice,\n    )\n    if response.status != \"completed\":          # anything else is an outcome, not data\n        return Outcome(\"rejected\", None, [f\"response status {response.status}\"])\n    if response.output_parsed is None:          # no parsed output: treat as a refusal\n        return Outcome(\"rejected\", None, [\"no parsed output; treat as a refusal\"])\n    return accept(response.output_parsed.model_dump_json(), source_text, expected_invoice_id)\n\n123456789101112131415\n\n# Documentation-derived; not executed against a paid API for this tutorial.def extract_constrained(client, model: str, source_text: str, expected_invoice_id: str) -> Outcome:    response = client.responses.parse(        model=model,        input=[            {\"role\": \"system\", \"content\": \"Extract the invoice fields. Use null when the text does not state a value.\"},            {\"role\": \"user\", \"content\": source_text},        ],        text_format=Invoice,    )    if response.status != \"completed\":          # anything else is an outcome, not data        return Outcome(\"rejected\", None, [f\"response status {response.status}\"])    if response.output_parsed is None:          # no parsed output: treat as a refusal        return Outcome(\"rejected\", None, [\"no parsed output; treat as a refusal\"])    return accept(response.output_parsed.model_dump_json(), source_text, expected_invoice_id)\n```\n\nThe call is the one OpenAI’s [structured-output guide](https://developers.openai.com/api/docs/guides/structured-outputs) documents for the Python SDK as of October 2026: `client.responses.parse(...)` with `text_format=Invoice`, which accepts the Pydantic model directly and returns `output_parsed` as an instance of it. The guide’s examples name `gpt-6-astra`; pin whichever model you run and record it. I wrote this adapter from the documentation and exercised it with a mocked client in the fixtures below, not against a paid API, and I’d rather say that than imply a live result.\n\nTwo lines do the work the decoder can’t. The status check runs before anything reads the object: a response that isn’t `completed` is routed, not used. A completed response with no parsed output is a refusal, not an empty invoice. Then the object goes through `accept()` unchanged, because the decoder enforced the shape and only the shape; the constraint guarantees that `total_cents` is an integer, not that it’s 125000 rather than 12500. Keep the contract plain too, since not every Pydantic feature survives the trip into a provider’s schema subset.\n\n## Run Both Paths Through the Same Failure Cases\n\nThe fixture suite turns the sentences above into fourteen test cases, from a missing required key and a code fence around the JSON to a fabricated quote, a refusal and exhaustion with no fourth call. Four of them, with the mocked client, look like this:\n\n``` python\nimport unittest\nfrom types import SimpleNamespace\n\nSOURCE = \"INVOICE 2026-0417\\nBill to: Northwind Traders\\nTotal due: EUR 1,250.00\\nDue 30 days from issue.\"\nID = \"2026-0417\"\nGOOD = {\"invoice_id\": ID, \"currency\": \"EUR\", \"total_cents\": 125000, \"total_quote\": \"Total due: EUR 1,250.00\"}\n\nclass Fixtures(unittest.TestCase):\n    def test_schema_valid_wrong_amount_is_rejected(self):\n        o = accept(json.dumps({**GOOD, \"total_cents\": 12500}), SOURCE, ID)\n        self.assertEqual(o.status, \"rejected\")\n\n    def test_exhaustion_makes_no_fourth_call(self):\n        calls = []\n        def gen(src, schema, feedback):\n            calls.append(1); return \"not json at all\"\n        outcome, attempts = extract_with_retries(gen, SOURCE, ID)\n        self.assertEqual((outcome.status, attempts, len(calls)), (\"rejected\", 3, 3))\n\n    def test_wrong_amount_is_not_retried(self):\n        calls = []\n        def gen(src, schema, feedback):\n            calls.append(1); return json.dumps({**GOOD, \"total_cents\": 12500})\n        outcome, attempts = extract_with_retries(gen, SOURCE, ID)\n        self.assertEqual((outcome.status, len(calls)), (\"rejected\", 1))\n\n    def test_incomplete_response_is_not_data(self):\n        resp = SimpleNamespace(status=\"incomplete\", output_parsed=Invoice(**GOOD))\n        client = SimpleNamespace(responses=SimpleNamespace(parse=lambda **kw: resp))\n        self.assertEqual(extract_constrained(client, \"m\", SOURCE, ID).status, \"rejected\")\n\n12345678910111213141516171819202122232425262728293031\n\nimport unittestfrom types import SimpleNamespace SOURCE = \"INVOICE 2026-0417\\nBill to: Northwind Traders\\nTotal due: EUR 1,250.00\\nDue 30 days from issue.\"ID = \"2026-0417\"GOOD = {\"invoice_id\": ID, \"currency\": \"EUR\", \"total_cents\": 125000, \"total_quote\": \"Total due: EUR 1,250.00\"}  class Fixtures(unittest.TestCase):    def test_schema_valid_wrong_amount_is_rejected(self):        o = accept(json.dumps({**GOOD, \"total_cents\": 12500}), SOURCE, ID)        self.assertEqual(o.status, \"rejected\")     def test_exhaustion_makes_no_fourth_call(self):        calls = []        def gen(src, schema, feedback):            calls.append(1); return \"not json at all\"        outcome, attempts = extract_with_retries(gen, SOURCE, ID)        self.assertEqual((outcome.status, attempts, len(calls)), (\"rejected\", 3, 3))     def test_wrong_amount_is_not_retried(self):        calls = []        def gen(src, schema, feedback):            calls.append(1); return json.dumps({**GOOD, \"total_cents\": 12500})        outcome, attempts = extract_with_retries(gen, SOURCE, ID)        self.assertEqual((outcome.status, len(calls)), (\"rejected\", 1))     def test_incomplete_response_is_not_data(self):        resp = SimpleNamespace(status=\"incomplete\", output_parsed=Invoice(**GOOD))        client = SimpleNamespace(responses=SimpleNamespace(parse=lambda **kw: resp))        self.assertEqual(extract_constrained(client, \"m\", SOURCE, ID).status, \"rejected\")\nRan 17 tests in 0.002s\n\nOK\n\n123\n\nRan 17 tests in 0.002s OK\n```\n\nRead what the mock proves and what it doesn’t. It proves the routing: a wrong amount is rejected on the first attempt and never retried, exhaustion returns after exactly three calls, an incomplete response never reaches the source checks, and a refusal stops after one. It proves nothing about how often a real model produces the 12,500-cent total, and nothing about cost or latency; there are no percentages in this article because a success rate from canned responses would describe the fixtures, not a model. When you run the suite against a real adapter, record the model name, the SDK version, the outcome class and the attempt count per case, mask the source text in the logs, and report parsed, structurally valid and correct-against-the-source as three separate columns, with `needs_review` in the third as not yet correct.\n\n## Let Only Accepted Results Reach the Next Step\n\nThe pipeline consumes three outcomes and treats them differently. `accepted` goes downstream. `needs_review` goes to a person with the candidate and the reasons attached. `rejected`, whether from a contradiction, a refusal or an exhausted budget, goes to an error path that records what happened and does nothing else; a retry never repeats a write, and a repair attempt never acquires permissions the first attempt didn’t have.\n\nPrefer constrained generation when the provider supports your contract, because it removes the shape failures and the retries they’d cause; fall back to the prompt-only loop with the same `accept()` when it doesn’t. In both cases I’d keep the typed contract and the source check even though the generator already enforces structure, because structure is the one thing it can enforce and correctness is the thing you need. The deployment test is a single question you can ask of a pipeline you already run: does every failure path stop before the downstream action? If the evidence is missing, the response is incomplete or the three attempts are spent, the result has to be an explicit review or error outcome, not a plausible object that happened to parse.", "url": "https://wpnews.pro/news/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained", "canonical_source": "https://machinelearningmastery.com/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained-decoding/", "published_at": "2026-11-12 11:00:00+00:00", "updated_at": "2026-11-12 13:21:19.724543+00:00", "lang": "en", "topics": ["large-language-models", "ai-tools", "structured-data", "developer-tools"], "entities": ["OpenAI", "Pydantic", "vLLM", "xgrammar", "Responses API", "Northwind Traders"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained", "markdown": "https://wpnews.pro/news/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained.md", "text": "https://wpnews.pro/news/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained.txt", "jsonld": "https://wpnews.pro/news/getting-reliable-structured-outputs-from-llms-validation-retries-and-constrained.jsonld"}}