{"slug": "pydantic-ai-structured-outputs-and-evals-on-bedrock", "title": "Pydantic AI structured outputs and evals on Bedrock", "summary": "Pydantic AI has released structured outputs and evaluation tools for Amazon Bedrock, enabling developers to constrain LLM outputs to typed schemas at the decoder level and assess content quality with Pydantic Evals. The approach eliminates retry loops and parsing glue code by guaranteeing JSON output matches a Pydantic BaseModel schema, then layers content validation and LLM-as-judge evaluation for open-ended responses.", "body_md": "# Pydantic AI structured outputs and evals on Bedrock\n\nWhen an LLM first breaks in production it’s rarely dramatic. It returns valid JSON, your parser is happy, and then three services downstream something falls over because `passengers`\n\ncame back as the string `\"two\"`\n\ninstead of the integer `2`\n\n. Nothing threw. Nothing logged an error. The shape was right and the meaning was wrong, and you spend an afternoon working out which one of those it was.\n\nI’ve lost that afternoon more than once, which is roughly why I now think about trusting a model in stages. You lock down the shape of what comes back first. Then you ask whether the content inside that shape was any good. Then you find a way to judge the open-ended parts that no exact-match assertion is ever going to cover. [Pydantic AI](https://ai.pydantic.dev/) and [Pydantic Evals](https://ai.pydantic.dev/evals/) cover that whole span, and a lot of the moving parts have landed or matured in the last few months, so it’s worth walking end to end.\n\n```\n1. shape                 2. content              3. judgement\n   structured    ──►         Pydantic Evals ──►      LLM-as-judge\n   outputs                   cases +                 calibrated\n   (constrained              evaluators              against your\n    decoding)                                        own labels\n   ───────────               ──────────────          ─────────────\n   is it the                 is the content          was the open\n   right shape?              correct?                text any good?\n```\n\n## Why structured outputs beat raw JSON [#](#why-structured-outputs-beat-raw-json)\n\nGetting structured data out of a model used to mean one of a few hacks, and all of them leak.\n\nThe first is prompt-and-pray. You ask for JSON and you get back valid-looking JSON wrapped in a markdown fence, or with a cheerful sentence of preamble in front of it, or with a trailing comma that kills your parser. So you write a parse step, then a retry loop, then a little function that strips fences, and now you own a pile of glue code whose only job is to apologise for the model.\n\nJSON mode is the next step up. It guarantees the output is syntactically valid JSON, which is better, but it’s a low bar. Nothing in JSON mode stops the model returning the wrong fields, the wrong types, or a number where you asked for a string. You still need validation, you still get failures, you’ve just pushed them one step downstream.\n\nThen there’s the tool-calling trick: you define a function whose parameters are your schema and let the model “call” it. It works, but it’s a semantic mismatch, because you’re bending tool-invocation plumbing into a data-extraction job and carrying the tool-calling overhead for something that was never a tool.\n\nStructured outputs fix this at the decoder, where it should be fixed. The provider compiles a grammar from your schema and constrains token generation to it, so the output is guaranteed to parse into the schema rather than usually parsing into it. That takes out the retry loops and the fence-stripping glue, along with the whole class of bug where a downstream system trips over a field that came back as `null`\n\ninstead of `0`\n\n. You ask for a shape and that’s what you get.\n\n## How Pydantic AI fits in [#](#how-pydantic-ai-fits-in)\n\nYou hand it a `BaseModel`\n\nas the `output_type`\n\n, it derives the JSON schema, drives the provider’s structured-output mode, and validates the response straight back into the typed object. You get the decoder-level guarantee and a real Python object: type safety, IDE autocomplete, and your own field validators sitting there as a second line of defence. The model gives you the right shape, Pydantic gives you the right object, and the seam between the two more or less disappears.\n\nIf you’ve spent any time hand-rolling `try/except json.loads`\n\nwith a retry counter, the difference is hard to oversell. The boilerplate doesn’t shrink, it stops existing. Here’s the before, a version I’ve written more than once:\n\n```\nraw = call_model(prompt)\ntry:\n    booking = Booking(**json.loads(strip_fences(raw)))\nexcept (json.JSONDecodeError, ValidationError):\n    raw = call_model(prompt + \"\\n\\nReturn ONLY valid JSON.\")  # ask nicely, again\n    booking = Booking(**json.loads(strip_fences(raw)))         # and hope\n```\n\nAnd the after:\n\n```\nagent = Agent(\"bedrock:anthropic.claude-sonnet-4-5-20250929-v1:0\", output_type=Booking)\nbooking = agent.run_sync(prompt).output  # already a typed Booking, guaranteed\n```\n\nAll of that glue is gone, and the guarantee moved into the decoder where it belongs.\n\n## Structured outputs on Bedrock [#](#structured-outputs-on-bedrock)\n\nIf you’re AWS-native like I am, the development worth knowing about is that this stopped being a client-side trick on Bedrock.\n\nAWS shipped [structured outputs natively on Amazon Bedrock](https://aws.amazon.com/about-aws/whats-new/2026/02/structured-outputs-available-amazon-bedrock) on the 4th of February 2026, and the mechanism is the one I described above: [constrained decoding for schema compliance](https://aws.amazon.com/blogs/machine-learning/structured-outputs-on-amazon-bedrock-schema-compliant-ai-responses), with no prompt engineering or extra checks bolted on the side. There are two ways in. You define a JSON schema for the response format, or you use strict tool definitions so the model’s tool calls match your spec. It went generally available for Anthropic’s Claude 4.5 models and a set of open-weight models, across the [Converse, ConverseStream, InvokeModel and InvokeModelWithResponseStream APIs](https://docs.aws.amazon.com/bedrock/latest/userguide/structured-output.html), and reached [GovCloud (US)](https://aws.amazon.com/about-aws/whats-new/2026/04/amazon-bedrock-structured-outputs-govcloud) on the 1st of April 2026.\n\nBefore this the Converse API left you reaching for function calling as the workaround, the same semantic mismatch as everywhere else. Now the guarantee comes from Bedrock itself. For a Pydantic AI stack running on Bedrock, the constrained decoding happens server-side and Pydantic AI just validates what comes back, with nothing faked in between.\n\n## A valid shape doesn’t mean good content [#](#a-valid-shape-doesnt-mean-good-content)\n\nA guaranteed schema tells you the response is well-formed. It tells you nothing about whether the content is correct, relevant, or actually true. The model can hand you a beautifully typed object full of confident nonsense and your validators will wave it straight through, because every field is exactly the type you asked for.\n\nThat’s the gap evals fill, and Pydantic Evals is the harness for it.\n\n## The three core abstractions [#](#the-three-core-abstractions)\n\nThe framework rests on three abstractions, and you’ll have them memorised in about five minutes.\n\nA **Case** is one test: an input, an optional expected output, and some metadata. Think of it as a single row in a test table. A **Dataset** is a collection of Cases you run together. An **Evaluator** is the thing that looks at an output and decides pass or fail, or hands back a score.\n\nYou write a `task`\n\nfunction that wraps your agent, call `dataset.evaluate(task)`\n\n, and it runs every Case, applies the evaluators, and gives you back a report with pass rates and scores. If you’ve written a parametrised pytest before, none of this will throw you, and that familiarity is the best thing about it.\n\n## The evaluators you get for free [#](#the-evaluators-you-get-for-free)\n\nA handful ship in the box and they cover more ground than you’d expect:\n\n`EqualsExpected`\n\n,`Equals`\n\n,`Contains`\n\nfor exact and substring matching, for the cases where the answer really is deterministic.`IsInstance`\n\nchecks the output is the right Pydantic model or type, so it slots straight into a stack that’s already validating shape.`MaxDuration`\n\ngives you a latency budget. A correct answer that takes nine seconds is still a failure in production.`LLMJudge`\n\ntakes a rubric and grades the output with an LLM, either binary pass/fail or a 0 to 1 score, with a swappable judge model.\n\nCustom evaluators are just a class that returns a bool, an int or float, or a string. The bool is an assertion, the number is a score, the string is a label. That’s the entire contract, and it’s enough to build whatever the built-ins don’t cover.\n\nPut together, a small suite reads about how you’d hope:\n\n``` python\nfrom pydantic_evals import Case, Dataset\nfrom pydantic_evals.evaluators import IsInstance, LLMJudge\n\ndataset = Dataset(\n    cases=[\n        Case(\n            name=\"refund_within_window\",\n            inputs=\"bought this yesterday, I want my money back\",\n        ),\n    ],\n    evaluators=[\n        IsInstance(type_name=\"SupportReply\"),\n        LLMJudge(rubric=\"Reply approves the refund and stays civil about it\"),\n    ],\n)\n\nreport = dataset.evaluate_sync(handle_ticket)\nreport.print(include_input=True, include_output=True)\n```\n\nThe shape check and the judgement check sit side by side, which is the whole point. One asserts the response is the right thing. The other asks whether it was any good.\n\n## Handling non-determinism [#](#handling-non-determinism)\n\nRun the same prompt twice and you can get two different answers even at temperature zero, because the [inference stack isn’t batch-invariant](https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/) and you don’t control the batch size on a hosted API. A single pass or fail tells you almost nothing.\n\nPydantic Evals handles this with a `repeat`\n\nparameter. Set `repeat=5`\n\nand each Case runs five times, and you get back a pass rate instead of a flaky single result. That one flag is the difference between a suite you trust and one that goes red on a Tuesday for reasons nobody can reconstruct. There’s also Tenacity-based retry handling for transient failures, so a rate limit doesn’t take down the whole run.\n\nPick your pass-rate threshold deliberately. You almost never want 100%. You want a number that reflects how much variation your users will tolerate and how bad a failure actually is. That’s a product call as much as a technical one, and it deserves more thought than just reaching for the round number.\n\n## LLM-as-judge [#](#llm-as-judge)\n\n`LLMJudge`\n\nis where this gets useful, because most real outputs can’t be graded by exact match. Was the summary faithful to the source? Was the tone right? Did it answer the question that was asked? That needs judgement, and a second model can supply it. This is the LLM-as-judge pattern, and it’s the natural next step after structured outputs and basic evals.\n\nAn LLM judge is worth nothing until you’ve checked it agrees with a human. Prefer binary pass/fail over a 1 to 5 scale, because nobody can tell you what separates a 3 from a 4 and your reviewers won’t agree on it either. Use a strong judge model, ideally from a different family than the one you’re grading, so you’re not just measuring a model’s fondness for its own output. And [calibrate against your own labels](https://hamel.dev/blog/posts/llm-judge/) before you trust a single number it produces. If the judge and a human disagree half the time, the score tells you almost nothing about quality.\n\nUsed well, it’s a quality gate for the parts of the response structured outputs can’t reach. Used lazily, it’s theatre, and the framework can’t tell the difference. It gives you the mechanism, and the calibration is on you. I went the long way round on that [building lgtmaybe](/posts/building-lgtmaybe/), where an uncalibrated reviewer produced confident findings nobody could trust.\n\n## Asserting on the trace [#](#asserting-on-the-trace)\n\nWith the [Logfire](https://logfire.pydantic.dev/docs/) extra installed, Pydantic Evals captures OpenTelemetry spans while your task runs. `HasMatchingSpan`\n\nthen lets you assert against the trace itself: which tools were called, in what order, with what arguments. Not just whether the final answer was right, but whether the agent took a sane path to get there.\n\n```\nCase input\n    │\n    ▼\ntask function ──► Agent run ──┬──► tool call: search ──┐\n                             ├──► tool call: fetch  ──┤\n                             └──► final output         │\n                                     │                 │\n                    ┌────────────────┘                 │\n                    ▼                                   ▼\n          output evaluators                    span evaluators\n          IsInstance, LLMJudge                 HasMatchingSpan\n                    │                                   │\n                    └─────────────────┬─────────────────┘\n                                      ▼\n                              EvaluationReport\n```\n\nFor anything agentic, MCP tool calls or multi-step retrieval, output-only evals lie to you constantly. An agent can stumble onto the right answer through a completely broken trajectory, and you won’t know until it hits an input where the luck runs out. Span evaluation catches that class of failure, and I add it to any agent I’d actually ship.\n\n## What it doesn’t do [#](#what-it-doesnt-do)\n\nIt’s a regression harness, not a platform, and the gaps are worth naming so you don’t go looking for things that aren’t there.\n\nThere’s no built-in metric library, no G-Eval, no RAGAS-style faithfulness scoring out of the box. No red-teaming. No pass@k, confidence intervals, or statistical significance testing baked in. Judge bias mitigation is manual. The docs are honest that the LLM judge carries length and style biases, and the mitigation is yours to run: temperature zero, multiple judges, and validating against human labels.\n\nI don’t hold any of that against it. It’s code-first, type-safe, Pydantic-native, backend-agnostic, and the library itself is free. Logfire is the optional paid backend if you want somewhere for the traces to land. If you need a wider metric library or a production dashboard, pair it with something like [DeepEval](https://deepeval.com/) or [Langfuse](https://langfuse.com/) rather than expecting Pydantic Evals to be those things. It does one job, turning stochastic output into a number you can gate a merge on, and it does it without dragging a platform into your repo.\n\n## Where this leaves you [#](#where-this-leaves-you)\n\nSo the picture holds together. Structured outputs, now native on Bedrock, guarantee the shape. Pydantic AI turns that shape into a typed object you can actually work with. Evals tell you whether the content inside the shape was any good. And LLM-as-judge, once you’ve calibrated it, scores the open-ended parts exact match was never going to reach.\n\nIf you’re already on Pydantic AI with validators in place, none of this is a rewrite. It’s the next few steps on the same path, and most of them are a few lines of code each.", "url": "https://wpnews.pro/news/pydantic-ai-structured-outputs-and-evals-on-bedrock", "canonical_source": "https://coles.codes/posts/pydantic-evals/", "published_at": "2026-07-14 00:00:00+00:00", "updated_at": "2026-07-14 05:54:37.667601+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-tools", "developer-tools"], "entities": ["Pydantic AI", "Amazon Bedrock"], "alternates": {"html": "https://wpnews.pro/news/pydantic-ai-structured-outputs-and-evals-on-bedrock", "markdown": "https://wpnews.pro/news/pydantic-ai-structured-outputs-and-evals-on-bedrock.md", "text": "https://wpnews.pro/news/pydantic-ai-structured-outputs-and-evals-on-bedrock.txt", "jsonld": "https://wpnews.pro/news/pydantic-ai-structured-outputs-and-evals-on-bedrock.jsonld"}}