{"slug": "how-to-evaluate-an-ai-agent", "title": "How To Evaluate An AI Agent", "summary": "Pydantic AI introduced Pydantic Evals, an evaluation library designed to test AI agents by combining deterministic structural checks with LLM-based judgment for subjective qualities. The library includes evaluators such as IsInstance for type validation and LLMJudge for rubric-based scoring, addressing the challenges of non-deterministic outputs and subjective criteria in agent testing.", "body_md": "In classical test-driven development, we deal with deterministic outcomes. We write assertions against values and types we already know in advance — `assert result == expected`\n\n, and we move on.\n\nAI agents don't play by those rules. Because the underlying LLM is non-deterministic, giving it the same input twice can produce two differently-worded outputs. That alone breaks the classical assertion model.\n\nBut there's a second problem, beyond just consistency. Say we have a customer support agent, and we want it to be helpful, empathetic, professional, and friendly when it talks to customers. Even if the output *were* consistent, how do we test for qualities like that? There's no fixed value or type to assert against — \"empathetic\" isn't a type or an exact string, it's a subjective judgment call.\n\nSo we're dealing with two distinct problems: unpredictable output, and qualities that are inherently subjective. Testing an AI agent means solving for both.\n\n[Pydantic Evals](https://pydantic.dev/docs/ai/evals/evals/) is Pydantic AI's recommended way to test the output of your AI agents. It's an evaluation library that gives you a set of evaluators purpose-built for exactly this kind of testing — some for the deterministic parts of your agent's behavior, and some for the parts that require judgment.\n\nNot everything about an AI agent is unpredictable. If your agent is set up with a structured output type, you can — and should — still test that the shape of the response is correct, even if the *content* varies.\n\nFor example, say we have an agent that returns a structured output type:\n\n``` python\nfrom pydantic_ai import Agent\nfrom dataclasses import dataclass\n\n# Response Model\n@dataclass\nclass Response:\n    message: str\n    sentiment: str\n\n# Agent Setup\nsupport_agent = Agent(\n    model=\"openai/gpt-5.5\",\n    name=\"customer_support_agent\",\n    instructions=\"\"\"\n        You are a customer support agent for an online store.\n    \"\"\",\n    output_type=Response\n)\n```\n\nWe can define a test case that checks the output is an instance of that `Response`\n\ntype, using the `IsInstance`\n\nevaluator:\n\n``` python\nfrom pydantic_evals import Case\nfrom pydantic_evals.evaluators import IsInstance\n\ntest_case = Case(\n    name=\"check_agent_response\",\n    inputs=\"where is my order?\",\n    evaluators=[\n        IsInstance(type_name='Response')\n    ]\n)\n```\n\nThis is a classic assertion in spirit — we know exactly what type we expect back, so we test for it directly. No judgment call required.\n\nStructural checks like `IsInstance`\n\ndon't help us with the harder problem: was the response actually *good*? Was it empathetic? Professional? Did it address the customer's question?\n\nThis is where `LLMJudge`\n\ncomes in. It uses an LLM as a judge to evaluate your agent's output against a set of criteria you define — scoring qualitative aspects of the response that a type check could never catch.\n\n``` python\nfrom pydantic_evals.evaluators import LLMJudge\n\nquality_case = Case(\n    name=\"check_response_quality\",\n    inputs=\"where is my order?\",\n    evaluators=[\n        LLMJudge(\n            rubric=\"Response should be empathetic, professional, and directly address the customer's question about their order status.\"\n        )\n    ]\n)\n```\n\nUnder the hood, `LLMJudge`\n\nsends your agent's output and your rubric to a judge model, which returns a score (and often a reason) reflecting how well the response meets your criteria. Rather than asserting on an exact value, you're asserting on a *standard*.\n\nTogether, `IsInstance`\n\nand `LLMJudge`\n\ncover both problems from earlier: structural correctness for the deterministic parts, and rubric-based scoring for everything else.\n\nI go deeper into `LLMJudge`\n\n— including setting up rubrics, reading scores, and combining multiple evaluators in a single test suite — in this video, part of the Master Pydantic AI series on my [YouTube channel](https://youtube.com/@joxiahdev).\n\nIf you find it useful, I'd really appreciate a like and a subscribe on the channel.", "url": "https://wpnews.pro/news/how-to-evaluate-an-ai-agent", "canonical_source": "https://dev.to/joxiahdev/how-to-evaluate-an-ai-agent-1oel", "published_at": "2026-08-04 08:00:00+00:00", "updated_at": "2026-08-04 08:15:17.571910+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models", "ai-tools"], "entities": ["Pydantic AI", "Pydantic Evals", "LLMJudge", "IsInstance", "OpenAI"], "alternates": {"html": "https://wpnews.pro/news/how-to-evaluate-an-ai-agent", "markdown": "https://wpnews.pro/news/how-to-evaluate-an-ai-agent.md", "text": "https://wpnews.pro/news/how-to-evaluate-an-ai-agent.txt", "jsonld": "https://wpnews.pro/news/how-to-evaluate-an-ai-agent.jsonld"}}