Artificial Intelligence #
Teams building AI agents on AWS use a range of frameworks, including LangGraph, LangChain, Strands Agents, and Amazon Bedrock AgentCore. Each framework records agent execution in its own way, so the structure of messages, tool calls, and handoffs between agents differs from one to the next. Evaluation tooling tends to follow the same boundaries: teams build metrics, scoring logic, and result storage against a single framework’s trace format, and they must rebuild them when they switch frameworks or run agents on more than one.
This coupling makes a basic development question harder to answer: did a change to the prompt, model, or tools improve the agent? Answering it requires consistent metrics across versions of an agent and a shared record of past results to compare against. Agent quality also spans several dimensions that a single accuracy score does not capture: tool calling, response quality, responsible AI, performance, multi-turn conversation, multi-agent coordination, and reasoning.
In this post, we introduce the Universal Agent Evaluation Framework (UAEF). Customers across 12 organizations in the telecom, healthcare, and financial services industries used UAEF to evaluate their AI agents with one consistent set of metrics and compare agent versions before release. UAEF is a framework-agnostic evaluation system: it uses adapters to convert traces from each framework into a common format, then scores those traces with 36 built-in metrics across 7 dimensions. It stores the results in Amazon DynamoDB and Amazon Simple Storage Service (Amazon S3) and generates reports and insights that inform the next iteration of the agent. This post describes how UAEF works, provides a playbook for choosing metrics and preparing datasets, and points to runnable notebooks for each supported framework.
UAEF: One evaluation layer for agent frameworks #
UAEF is a framework-agnostic evaluation system for AI agents. It takes the agent a team already has and scores it the same way across supported frameworks and runtimes. UAEF takes two inputs: the agent under test and a dataset to run it on. The dataset provides the inputs for each test case, and it can also include ground-truth answers, the expected output for each input. Ground truth is optional, and only metrics that compare the agent’s response against a reference use it. To evaluate tool calling, the ground truth must also include the expected tool calls for each test case: which tools the agent should call, in what order, and with what arguments. Everything after that happens inside the UAEF.
Figure 1 shows the path a single evaluation takes, from the inputs to the reports and insights. First, an adapter reads the agent’s native execution, its messages, tool calls, and agent-to-agent handoffs, and normalizes it into one canonical trace, so that the steps that follow are identical no matter which framework produced the agent. Next, UAEF screens the trace for personally identifiable information (PII), such as email addresses, phone numbers, and credit card numbers, before data is stored. The evaluation engine then scores the trace on a selection from 36 built-in metrics across 7 quality dimensions. When you persist a run, UAEF records it as an experiment in Amazon DynamoDB and Amazon S3: Amazon DynamoDB holds the per-experiment metadata used for listing and comparison, and Amazon S3 holds the full results. With runs stored, a team can compare an agent version against a baseline to detect regressions and generate reports and insights that explain what changed and why. Those reports close the loop: the team evaluates the agent, learns what regressed or improved, changes the prompt, model, or tools, and evaluates again.
The rest of this section follows that flow in order: the adapters that make UAEF framework-agnostic, the evaluation engine and its metrics, PII detection, experiment storage and comparison, and the reports that close the loop.
Canonical trace and adapters
Adapters are what make UAEF framework-agnostic. An adapter reads the record of an agent’s run, including its messages, tool calls, and agent-to-agent handoffs. It converts that record into one canonical representation called the AgentTrace, which is the only thing the rest of UAEF scores. Because of that conversion, UAEF works with the agent, whatever SDK it was built with (such as LangGraph, LangChain, or Strands Agents). It can run locally, behind an HTTP endpoint, or on a managed runtime such as Amazon Bedrock AgentCore, with traces in Langfuse or not captured at all. UAEF provides an adapter for each of these, and they produce the same AgentTrace, so metrics, scores, and comparisons stay consistent across every one of them.
A trace reaches UAEF two ways. If one already exists, from a past run, a log, or an observability store, it is handed to UAEF and the matching adapter normalizes it. If not, UAEF invokes the agent for you: given a runtime Amazon Resource Name (ARN) or an HTTP endpoint, it calls the agent on each dataset input, collects the response, and adapts it in one step. Either way, the output is the same AgentTrace.
Because every metric, mode, and report operates on the AgentTrace rather than on a framework native format, the scoring that follows is identical no matter where the trace came from. That separation, normalization first and scoring second, is what lets a team compare an agent built on one framework against an agent built on another on a single scale.
Evaluation engine
The evaluation engine scores a canonical trace across 7 quality dimensions, each capturing a distinct aspect of agent behavior that a single accuracy number would hide. Tool calling measures whether the agent selected the right tools, called them in a sensible order, and passed valid arguments. Response quality measures whether the final answer is relevant, complete, accurate, and grounded in the provided context rather than hallucinated.
Responsible AI measures safety, bias, toxicity, and resistance to prompt injection. Performance measures operational cost in latency, token use, and throughput. Multi-turn measures how well the agent sustains a conversation, including context retention, coherence, and goal completion across turns. Multi-agent measures coordination in systems of cooperating agents, such as task delegation and the division of work. Reasoning measures the internal logic of the agent’s chain of thought, including consistency and freedom from fallacies.
Each dimension is computed by one or more of 36 built-in metrics, and every metric returns a normalized score in the interval [0, 1], where 1 is best, so dimensions and the overall result share one scale. Metrics come in two families. Deterministic metrics compute a score directly from the trace and the ground truth and reproduce exactly on repeated runs. Large language model (LLM)-as-a-judge metrics score qualities that resist a closed-form definition, such as response relevance or safety, by prompting a judge model on Amazon Bedrock. You can also add your own custom metrics or draw on the RAGAS and DeepEval integrations.
Because judge metrics feed an agent’s own output back into a prompt, that content is treated as untrusted input: an agent that can influence its judge can influence its own scores. The rubric is sent separately from the content, and the content is escaped so it cannot break out of its delimiter and be read as instructions. Each judge response is parsed strictly and must cite a verbatim span of the content it scored. A score whose evidence is not found is rejected rather than recorded. Judge scores are evaluation signal, not a safety decision, and should not gate a security control. For runtime enforcement on the agent itself, apply Amazon Bedrock Guardrails in its own invocation path. The full model is in SECURITY.md.
Experiment management and comparison
UAEF records each evaluation as an experiment, so results from different agent versions can be compared later. Calling evaluate() or batch_evaluate() with persist=True creates a new experiment, identified by an optional name and objective, and writes it to two stores with complementary roles. Amazon DynamoDB holds one row per experiment with the metadata used for listing and comparison: evaluation count, the average score per metric, and an overall average. Amazon S3 holds a JSON file with the full evaluation results, including per-metric scores and the judge’s reasoning. Teams that handle sensitive data can turn on PII redaction for the stored copy.
The Python library also models a finer-grained structure. An experiment contains runs, and each run records the agent design under test: framework, model, tools, system prompt, and parameters. A run can also record a configuration snapshot and a Git commit. You can mark one run as the baseline for its experiment. This ties each score to the change that produced it, whether a new prompt, a different model, or an added tool.
Comparison works on the stored per-metric averages. For each metric that two runs have in common, the comparison engine calculates the absolute and percentage change. It then classifies the metric as a regression, an improvement, or unchanged, using a configurable threshold with a default of 5 percent. Several runs can be compared against a single baseline in one call. The regression detector also assigns a severity to each regression: minor (5–10 percent), moderate (10–20 percent), or severe (over 20 percent). It groups regressions by dimension, which shows, for example, whether a prompt change affected tool calling but not response quality.
In the web UI, the Compare tab lists stored experiments and filters them by framework or dataset. You select two or more experiments. The first is used as the baseline, and the UI shows per-metric scores with regressions and improvements highlighted. The UI calls the same library comparison engine, so results match what the Python API returns.
Report and insight generation
After an evaluation is stored, UAEF reports on it at three levels, each answering a different question. How did this run perform? Why did it score that way? How does it compare with earlier versions?
Results view. When an evaluation finishes, the web UI shows a summary of the experiment: the overall score, the number of test cases evaluated, the framework, and queries that failed to run. Per-metric averages appear with a dimension radar chart and a metric bar chart. A table lists every test case with its query, the agent’s response, the expected answer, and the score. Expanding a row shows the full response with its dimension and metric scores. For a multi-turn conversation, it shows each turn with its own scores and the tools the agent called.
Detailed report. From the results view or the experiment list, you can generate a statistical report on demand. UAEF loads the stored results from Amazon S3 and opens the report in a new browser tab. The report breaks results down by dimension and by metric. It shows confidence intervals and outliers for each metric, score distributions, correlations between metrics, a responsible AI summary, and test cases that reported failures. It closes with recommendations that point to the lowest-scoring dimensions and the metrics that vary most across test cases. The report is computed from the scores alone, without an LLM, so the same results produce the same report.
LLM-generated insights. In the Python library, you can pass a report to a model on Amazon Bedrock for analysis. For a single run, the analysis covers strengths, critical weaknesses, hypotheses about root causes, and prioritized recommendations. For a comparison of several runs, it covers the overall trend, likely causes of regressions and improvements, the risk each regression carries, and next steps. These insights suggest what to change in the prompt, model, or tools before the next evaluation.
Two ways to use it
You can run UAEF two ways over the same engine. With the Python library, you can evaluate agents in code from notebooks, scripts, and continuous integration and continuous delivery (CI/CD) pipelines. You can use the deployable AWS service to evaluate agents through a low-code web interface. You upload your dataset, select a framework and metrics, run the evaluation, and review the scores. Both paths run the identical evaluation engine and write to the same stores, so a result does not depend on which path produced it. You can start in the interface and move to code without changing how your results are scored or compared.
Evaluating agents with UAEF: A playbook #
Knowing what UAEF can measure is different from knowing what you should measure. A travel-booking agent and a policy Q&A agent fail in different ways, and scoring both on every dimension wastes time and Amazon Bedrock invocations. Use this playbook to pick the metrics that match your agent, then add more as new questions come up.
Getting started
UAEF is open source under the Apache 2.0 license in the awslabs/uaef GitHub repository. It includes the Python library, framework adapters, built-in metrics, AWS Cloud Development Kit (AWS CDK) infrastructure for the deployable service, the web interface, and runnable notebooks for each supported framework. The Installation Guide covers requirements, installing from a built package, and the development setup.
The service deployment guide covers deploying UAEF as a service with the web interface, the config reference, and teardown.
See it in action
A complete evaluation in the UAEF web interface, from ground-truth upload to per-dimension scores.
Choosing the metrics
Start from your scenario, not the metric list. Find the column that matches your agent and begin with the metrics checked in it. The Needs ground truth column shows which metrics compare against a reference answer or expected tool calls.
| Dimension | Metric | Needs ground truth | Scenarios | | | | | | | | | | | | RAG / knowledge Q&A | General Q&A | Tool-using agent | Multi-turn conversation | Reasoning | Safety and compliance | Multi-agent | Production monitoring | | Response Quality | | | | | | | | | | |