AI agent guardrails vs. evals: How to build more reliable agent systems

Arize head of developer relations Laurie Voss said that AI agent reliability depends on both guardrails and evals, which solve different engineering problems, after a live demo where a voice agent spoke over itself due to multiple response paths running simultaneously. Voss noted that as agents run longer and act with less supervision, failures can occur at many points, including tool calls, retry loops, and conflicting branches, making it essential to enforce guardrails at the code level while evals judge performance.

The voice agent knew what to say. But then it began saying three things at once. One response started, another cut in, and a third arrived before either of the first two had finished. The voices overlapped until the answer became impossible to follow. Laurie Voss, head of developer relations at Arize, watched the failure unfold during a live demonstration. The language model had understood the request, but the system had simply allowed several response paths to proceed simultaneously. “There was nothing wrong with the model,” Voss said. “There was definitely something wrong with the harness https://arize.com/resources/agent-harness-evaluation-tracing/ .” The incident captures a problem that will become harder to ignore as AI agents https://arize.com/glossary/agent/ run longer and act with less supervision. Model capability can carry an agent surprisingly far, but reliable behavior depends on the system that determines which tools it can use, what actions it can take, how its work is evaluated, and what happens after something goes wrong. That system needs both guardrails and evals https://arize.com/resources/ai-agent-evaluation/ . The terms often appear together, yet they solve different engineering problems. An hour-long agent run can fail in more places than a chatbot Early AI agents https://arize.com/glossary/agent/ required almost constant supervision. A developer might let one run for a few minutes, watch what happened, and intervene when it wandered off course. That autonomy window is expanding. Voss hears developers describe agents that can work for several hours, sometimes beginning with a product requirements document and returning with a substantial implementation. Whether every two-day coding run succeeds is less important than the direction of travel. Agents are being given more time, more tools, and more authority. A conventional chatbot interaction has a narrow failure surface. It receives a message and generates a response. A long-running agent https://arize.com/resources/agent-harness-evaluation-tracing/ may retrieve information, call several tools, modify files, retry failed operations, manage state, delegate work, and decide when the task is complete. Each new step creates another place where the system can deviate from the intended path: - A tool call https://arize.com/glossary/tool-calling/ can use the wrong arguments. - A retry loop can consume excessive time or tokens. - Two branches can attempt conflicting actions. - The agent can carry stale context into a later decision. - A valid intermediate action can produce a poor final outcome. - A useful final answer can conceal a wasteful or unsafe trajectory https://arize.com/resources/agent-harness-evaluation-tracing/ . For developers, longer trajectories create an orchestration and observability https://arize.com/resources/whats-an-agent-observability-platform/ problem. For product managers, greater autonomy changes the product contract. The team must decide how much authority the agent receives, when it needs approval, how it should recover, and what the user sees when the workflow leaves its expected path. Evals and guardrails solve different problems Voss draws a useful distinction between evals https://arize.com/glossary/evaluations/ and guardrails https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ . “The evals say what’s good and what’s bad, but the guardrail https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ s are enforced at the code level,” he explains. An eval https://arize.com/glossary/evaluations/ judges behavior while a guardrail constrains behavior. Here’s an easy way to break this down: Layer | The question it answers | Example | |---|---|---| Eval | Did the agent perform well? | Was the answer correct, grounded, useful, or complete? | Guardrail | Was the agent permitted to take that action? | Could it call this tool, send this message, or launch another response? | Harness | How does the entire run execute? | How are tools, state, limits, retries, permissions, and feedback coordinated? | The voice agent needed a guardrail https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ that permitted only one active answer. An eval https://arize.com/glossary/evaluations/ could detect the overlapping speech after the run, but detection alone would not prevent the behavior from recurring. Reliable systems translate product requirements into executable constraints. “Stay focused” is difficult to enforce. “Permit one active voice response at a time” can become a testable rule. A simplified policy might look like this: Illustrative pseudocode policy = AgentPolicy allowed tools={“knowledge search”, “calendar lookup”}, max parallel responses=1, max retries=2, approval required={“send email”, “make purchase”}, Prompt instructions still matter because they help shape the agent’s plan. Code-level controls provide a firmer boundary when the action carries cost, risk, or an irreversible consequence. Useful guardrails https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ can cover: - which tools the agent may invoke; - which credentials each tool receives; - how many actions may run concurrently; - how long a workflow may continue; - how many times a failed step may retry; - which actions require human approval https://arize.com/glossary/evaluation-gating/ ; - when the agent should stop, escalate, or ask for clarification. A high-quality final answer does not prove that the agent followed an acceptable path. Teams need to evaluate the outcome while also enforcing the boundaries around the process. Your AI judge may be grading yesterday’s world Guardrails can fail and so can evaluators https://arize.com/resources/llm-evaluation/ . Voss uses a financial-analysis agent in her tutorials. The agent researches recent events and produces a report. A generic correctness evaluator https://arize.com/guides/llm-as-a-judge/ then grades that report using its own internal knowledge. The result is a predictable mismatch. The agent cites information from the current day, while the evaluator https://arize.com/resources/llm-evaluation/ behaves as though the newer events have not happened. It marks the report incorrect every time because its frame of reference is stale. The evaluator appears authoritative because it produces a score. That score still reflects the context available to the judge. An eval https://arize.com/glossary/evaluations/ uator may need access to: - the current date and time; - the sources retrieved during the run; - the user’s actual goal; - the policies governing the product; - the tools available to the agent; - the complete trajectory https://arize.com/resources/agent-harness-evaluation-tracing/ ; - the product’s definition of a successful outcome. “Correctness” is rarely universal. A current financial report, a customer-support response, and a code migration each require different evidence. A generic judge https://arize.com/guides/llm-as-a-judge/ may evaluate fluency while missing the business rule that determines whether the answer is usable. Developers should test evaluators with the same skepticism they apply to the application. Product managers should define what evidence a judge needs before its score can influence a release decision https://arize.com/resources/the-eval-playbook-for-engineer-pm-collaboration/ . A failed run can become the next engineering brief The most useful eval output may be an explanation rather than a score. In her own side projects, Voss runs evals https://arize.com/glossary/evaluations/ that describe what went wrong. Her development environment pulls those explanations from Arize and gives them to the coding agent https://arize.com/blog/improve-ai-agents-traces-evals-harness/ responsible for the next iteration. The instruction is blunt: “Here are the explanations of all the things that went wrong in your last run. Fix yourself.” The workflow creates a bounded form of self-improvement: Agent run ↓ Traces and outputs ↓ Evals generate scores and explanations ↓ Failure explanations return to the IDE ↓ Coding agent proposes changes ↓ Tests and evals run again This loop does not require an agent to redesign and deploy itself without oversight. Evaluation feedback becomes a structured development task, which a coding agent https://arize.com/blog/improve-ai-agents-traces-evals-harness/ can act on before the next review. The distinction matters. A pass/fail result can tell a team that quality declined. An explanation can guide a specific change to: - a prompt; - a tool description; - a retrieval query; - a permission boundary https://arize.com/glossary/evaluation-gating/ ; - a retry policy https://arize.com/resources/agent-harness-evaluation-tracing/ ; - a piece of application code; - a regression test https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/ . The workflow also changes how teams think about production failures https://arize.com/resources/online-llm-evaluations/ . A bad run no longer has to remain an isolated incident in a trace viewer https://arize.com/resources/ai-agent-tracing-evaluation/ . It can become evidence for the next version of the system. Product requirements need operational counterparts Product teams often describe agent behavior using language that sounds clear during planning but becomes ambiguous during implementation. Consider a voice-booking agent. A requirement might say: The agent should stay focused on completing the booking. That sentence leaves several questions unresolved. Can the agent react to background speech? Should it answer unrelated questions during the booking? When has the task ended? What happens when the user changes direction halfway through? A more operational definition might specify that the agent must: - act only on speech directed toward the booking workflow; - maintain one active response at a time; - ignore unrelated background conversation; - request clarification when the destination is ambiguous; - stop after confirmation or transfer control to a human. These conditions can become guardrails https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ , eval criteria https://arize.com/resources/ai-agent-evaluation/ , or both. The product manager defines the intended experience. The developer converts that intent into system behavior. The eval reveals whether the experience occurred, while the guardrail https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ prevents actions that should never happen. Five questions to ask before increasing agent autonomy Before allowing an agent to run longer or act with less supervision, the team should be able to answer five questions. 1. What can the agent do without approval? List the permitted actions explicitly. Separate reversible work from actions that spend money, modify production data, communicate externally, or create legal obligations. 2. Which boundaries are enforced in code? Prompt instructions express desired behavior. Consequential restrictions should also have executable enforcement. 3. What context does the evaluator need? A judge cannot assess current information, domain policy, or tool use unless the relevant evidence is available during evaluation. 4. Can the team reconstruct the trajectory? A final answer may hide retries, unnecessary tool calls https://arize.com/blog/how-to-evaluate-tool-calling-agents/ , conflicting branches, or policy violations https://arize.com/glossary/evaluation-gating/ . Traces should expose the steps that produced the result. 5. How does failure information reach the next version? Decide whether failures become tickets, regression tests https://arize.com/resources/llm-evaluation/ci-cd-for-llm-apps/ , updated evals https://arize.com/resources/llm-evaluation/ , prompt changes, guardrail changes https://arize.com/glossary/evaluation-gating/ , or structured feedback for a coding agent https://arize.com/blog/improve-ai-agents-traces-evals-harness/ . Without that last connection, observability https://arize.com/glossary/agent-observability/ remains retrospective. The team learns what happened but does not systematically change what will happen next. The AI engineer becomes a systems engineer Voss defines an AI engineer https://arize.com/resources/what-is-ai-engineering/ as someone who brings substantial expertise from engineering or another domain, then uses AI effectively as another tool. That definition places judgment above novelty. The work requires more than knowing which model or framework was released last week. AI engineers increasingly need to reason across several layers at once: - model behavior - product intent orchestration https://arize.com/glossary/agent-orchestration/ - permissions guardrails https://arize.com/resources/guides/jailbreaking-ai-models/guardrails-for-llms/ evaluation design https://arize.com/resources/ai-agent-evaluation/ observability https://arize.com/glossary/agent-observability/ feedback and iteration https://arize.com/blog/improve-ai-agents-traces-evals-harness/ The language model may generate the answer, choose a tool, or propose a code change. The engineer builds the conditions under which those actions become dependable. As agents operate for longer periods, reliability will depend on systems that constrain what they can do, evaluate what they actually did, and carry useful evidence from one run into the next. Case in point via our original example: the model knew how to answer the question in the voice demo, but the engineering challenge was making sure only one answer reached the user and not three all at the same time .