AI Agent Evaluation: How to Test Workflows That Change Their Answers Umer Amin outlined a practical framework for evaluating AI agent workflows whose outputs vary between runs, arguing that exact-text-match tests produce brittle suites or false confidence. The approach defines each workflow's user goal, allowed actions, required evidence and unacceptable outcomes, then scores task completion, factual grounding, tool selection, policy adherence, escalation, latency and cost across a stable regression set and a separate fresh set. Amin recommends running evaluations across prompt, model, retrieval and tool changes, setting release thresholds for critical behaviors, and converting production incidents into new test cases. Traditional tests expect the same input to produce the same output. AI workflows can vary, so testing only exact text matches creates brittle suites—or false confidence when the expected phrase appears but the task failed. For each workflow, write the user goal, allowed actions, required evidence, and unacceptable outcomes. A support agent might need to answer from approved policy, cite the relevant source, and ask for help when the source is missing. Those conditions matter more than whether it uses a particular sentence. Collect realistic requests, edge cases, ambiguous inputs, and known failure examples. Include cases with missing context, conflicting instructions, malformed data, and tool errors. Keep a stable test set for regressions and a separate fresh set for discovering new problems. Score multiple dimensions: task completion, factual grounding, tool selection, policy adherence, escalation behavior, latency, and cost. Use deterministic checks where possible. For open-ended quality, use human review or a calibrated judge, then manually inspect disagreements. Run evaluations across prompt, model, retrieval, and tool changes. Include the whole path: input preparation, permissions, retries, side effects, and final handoff. A model can improve its answer while the workflow becomes less safe. Set release thresholds for critical behaviors. If a change drops a safety or task metric, stop rollout or send it to a limited test group. Track production feedback and turn incidents into new evaluation cases. Good evaluation is a loop: define success, test representative cases, review failures, and improve the system. It makes model variation measurable instead of mysterious. I share practical AI delivery notes at https://umer-amin.vercel.app/ https://umer-amin.vercel.app/ . How are you testing behavior when there is no single “correct” sentence?