# AWS UAEF: Universal Agent Evaluation Across 36 Metrics > Source: > Published: 2026-11-11 00:00:00+00:00 **On November 11, 2026, AWS published the [Universal Agent Evaluation Framework (UAEF)](https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/?ref=explainx) — an open evaluation layer that answers a question every agent team hits after the demo: did the prompt, model, or tool change actually improve behavior?** Instead of rebuilding metrics per SDK, UAEF **adapters** normalize LangGraph, LangChain, Strands, Bedrock agents, AgentCore, Langfuse, or custom JSON into one **AgentTrace**, then score the same 36 metrics whether the agent ran locally, behind HTTP, or on a managed runtime. The code lives on GitHub as **[awslabs/uaef](https://github.com/awslabs/uaef?ref=explainx)** under Apache 2.0. AWS says **12 customer organizations** in telecom, healthcare, and financial services used UAEF before the public post — a signal this is meant for release gating, not slideware. ## TL;DR | Piece | AWS (Nov 11, 2026) | |---|---| | Project | **UAEF** — Universal Agent Evaluation Framework | | License | **Apache 2.0** (`awslabs/uaef` ) | | Adapters | LangGraph, LangChain, Bedrock, Langfuse, **Strands** ,**AgentCore** , Generic JSON | | Metrics | **36** built-in across**7** dimensions | | Storage | **DynamoDB** metadata +**S3** full JSON per experiment | | Judges | Deterministic metrics + **Bedrock** LLM-as-judge with prompt-injection guards | | Surfaces | Python library, notebooks, deployable **web UI** , optional**EKS** API | ## The problem UAEF targets Teams on AWS already mix **[Strands decider patterns](https://explainx.ai/blog/amazon-strands-decider-internal-ticketing-sonnet-replacement-2026)**, **[Bedrock AgentCore](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026)**, and LangGraph graphs in different products. Each framework logs messages, tool calls, and handoffs differently, so **evaluation code rots per stack** — exactly the fragmentation [WorkflowArena-style orchestration benchmarks](https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026) try to measure from the outside. UAEF separates **normalization** from **scoring**: adapters emit AgentTrace; every metric consumes that shape. AWS claims you can compare an agent built on one framework against another on **one scale** — useful when a platform team standardizes gates but not SDK choices. ## Closed-loop flow AWS diagrams a repeatable loop: 1. **Adapter ingest** — pass an existing trace or let UAEF invoke the agent via ARN/HTTP and capture fresh output. 2. **PII screen** — detect emails, phone numbers, payment data before storage. 3. **Evaluate** — run selected metrics (subset or full 36). 4. **Persist** — optional experiment row in DynamoDB plus full results in S3. 5. **Compare & report** — highlight regressions vs baseline; optional LLM-written insights on what to change next. That maps directly to CI patterns explainx.ai recommends in [Raindrop reasoning memory benchmarks](https://explainx.ai/blog/raindrop-reasoning-agent-memory-tau2-bench-67-percent-2026) and [Infrathon infrastructure agent evals](https://explainx.ai/blog/infrathon-long-horizon-ai-infrastructure-agent-benchmark-2026) — except UAEF is **AWS-native storage** with a shipped UI. ## Metrics and judges Seven dimensions span behaviors a single accuracy number hides: | Dimension | What it catches | |---|---| | Tool calling | Wrong tool, bad argument shape, sequence errors | | Response quality | Relevance, completeness, hallucination vs context | | Responsible AI | Safety, bias, toxicity, prompt injection resistance | | Performance | Latency, token and cost efficiency | | Multi-turn | Retention, coherence, goal completion | | Multi-agent | Delegation, utilization, workflow completion | | Reasoning | Chain consistency, fallacies | Deterministic metrics replay identically; judge metrics call Bedrock with **untrusted-output handling** — rubric separated from content, escaped delimiters, evidence spans required or scores reject. AWS explicitly says judge scores are **not** a safety gate; use **[Bedrock Guardrails](https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html?ref=explainx)** on the runtime path for enforcement — same split we document in [our agent sandbox security post](https://explainx.ai/blog/claude-cage-vm-sandbox-claude-code-network-isolation-2026). Integrations include **RAGAS** and **DeepEval** metrics for retrieval-heavy agents. ## Experiment tracking builders actually need When `persist=True`, UAEF records framework, model, tools, prompts, optional git SHA, and per-metric averages. Mark a **baseline run** and the comparison engine flags regressions — the workflow MemCurrent-style memory papers chase with token counts, but for **full agent traces**. Multi-tenant note: AWS's **SECURITY.md** says UAEF expects **host-level tenant isolation** — separate stacks or accounts per customer — and scopes reads to Cognito `sub` inside a deployment. Do not bolt multi-customer SaaS onto one shared UAEF stack without that separation. ## How to start this week 1. `git clone https://github.com/awslabs/uaef` and`uv sync` per README. 2. Run notebook **00_langgraph.ipynb** or**06_agentcore.ipynb** against a toy agent. 3. Pick metrics from AWS's scenario table (RAG Q&A vs tool-using vs multi-turn columns in the blog). 4. Call `batch_evaluate(..., persist=True, experiment_name='v2 prompt')` before merging prompt changes. 5. Optional: deploy **uaef-service** with CDK when you want a reviewer UI on top of the same engine. AWS also documents dataset columns — `Question`, optional `Answer`, `expected_tool_calls` JSON, and `session_id` / `turn_id` for conversations — so exports from [Quick Skills agent catalogs](https://explainx.ai/blog/aws-quick-skills-agents-catalog-build-guide-2026) or internal ticket bots can reuse CSVs with minimal reformatting. ## Playbook highlights from the AWS post AWS's metric chooser table is the fastest onboarding doc: start from **scenario**, not the full metric list. Retrieval agents begin with faithfulness and context precision; tool agents need selection accuracy and parameter quality; compliance-heavy flows prioritize safety and injection detection. Pipeline choices are explicit: passive trace evaluation vs active `invoke_*` helpers, single trace vs batch, offline CI vs `OnlineEvaluator` on live traffic, ephemeral scores vs persisted experiments (only persisted runs compare). ## Deployment surfaces in more detail AWS documents three runnable surfaces that share one evaluation core: **Python library and notebooks.** The fastest path for a platform team is `evaluate()` on a trace captured from staging, then `batch_evaluate()` on a CSV exported from support tickets. Notebooks under `notebooks/` map frameworks to scenarios — LangGraph batch runs, AgentCore multi-agent sessions, Langfuse imports for production traces you already store. **Serverless web UI.** The `uaef-service` CDK stack provisions Cognito auth, API Gateway, worker Lambdas, Step Functions, DynamoDB, S3, and CloudFront for a React reviewer UI. AWS stresses that `cors_allowed_origins` is mandatory: blank means the deploy script derives the CloudFront origin on first UI deploy, while EKS mode requires an explicit origin because no distribution exists. **EKS API.** Teams that already run Bedrock inference on Kubernetes may prefer the containerized FastAPI deployment in `deploy_eks/` — same persistence schema, different ingress. Bring your own SSO and UI; keep experiment comparison on the Python API. Across modes, **only persisted experiments compare**. Ephemeral `evaluate()` calls help debug one bad trace; they will not show up in regression dashboards. ## Example metric bundles AWS recommends Rather than enabling all 36 metrics on day one, AWS's scenario table implies starter packs: | Agent shape | Start with | Why | |---|---|---| | RAG helpdesk | `answer_relevance` ,`hallucination_score` ,`ragas_faithfulness` | Catches wrong retrieval and invention without tool metrics | | Ticket router | `tool_selection_accuracy` ,`parameter_quality` ,`latency_score` | Measures wrong API and slow escalations | | Compliance bot | `safety_score` ,`prompt_injection_detection` ,`toxicity_score` | Separates policy incidents from helpfulness | | Pair programmer | `accuracy` ,`completeness` , reasoning chain metrics | Aligns with IDE agent regressions | | Swarm workflow | `delegation_quality` ,`workflow_completion` ,`coordination_efficiency` | Mirrors [WorkflowArena](https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026) failure modes | Add **multi-turn** metrics only when datasets include `session_id` and `turn_id` columns; otherwise UAEF treats rows as independent single-turn cases. ## CI/CD pattern A minimal release gate looks like this: 1. Export 50–200 production queries with redacted PII into a CSV using `PIIDetector` . 2. Mark the current production agent run as `baseline` in UAEF experiment metadata. 3. On each pull request that changes prompts, tools, or model ARNs, run `batch_evaluate(..., persist=True, experiment_name='pr-1234')` . 4. Fail the build if any dimension drops more than an agreed tolerance versus baseline averages. 5. Attach the HTML report link to the PR for human review of judge-reasoning fields stored in S3. This is the AWS-native version of patterns [Amazon's Bedrock evaluation playbook](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026) describes for DeepEval inside AgentCore — complementary, not redundant. ## Judge security details worth copying AWS's judge path treats agent output as **untrusted input**: rubric and content are separated, content is escaped, and judge answers must cite verbatim evidence spans or the score is rejected. That design responds to the same class of attacks [Claude memory exfiltration research](https://explainx.ai/blog/claude-memory-heist-web-fetch-exfiltration-ayush-paul-july-2026) documents — models that see their own evaluation prompt can game it. UAEF explicitly says **do not** use judge scores as runtime policy. Pair UAEF offline gates with Bedrock Guardrails or application-level tool allowlists on the live agent. ## Online evaluation versus offline batches AWS distinguishes **OfflineEvaluator** batch runs on frozen datasets from **OnlineEvaluator** hooks that score live conversations turn by turn. Offline runs belong in CI and release trains; online hooks belong in canary deployments where you sample a percentage of production traffic (with PII redaction) to catch drift when user phrasing shifts. The blog's **passive versus active** pipeline choice matters here: passive mode ingests traces your observability stack already captured (Langfuse, CloudWatch logs, custom JSON); active mode lets UAEF invoke agents so every test uses the same temperature and tool manifest. Mixed environments often passive-import staging traces but actively invoke production builds before promotion. ## Human-in-the-loop review UAEF documentation references **HITL** flows for disputed judge scores — important when LLM judges disagree with deterministic metrics on borderline safety cases. Treat HITL queues as part of the product, not an afterthought: without them, teams mute judge metrics and ship anyway. ## When not to adopt UAEF yet UAEF is heavy if you have **one** LangGraph demo and no baseline traffic. It pays off when: - Multiple teams ship agents on different frameworks under one platform org. - You already pay Bedrock judge costs and need shared storage. - Regulated customers ask for **experiment audit trails** . If you only need a single RAG faithfulness number, starting with RAGAS inside the app may be faster — then graduate to UAEF when regressions become cross-team. ## How UAEF fits next to other AWS agent posts - **[Bedrock tokenomics data lake](https://explainx.ai/blog/amazon-bedrock-tokenomics-data-lake-finops-2026)** — UAEF adds**quality** time series beside cost telemetry. - **[Trovata treasury agents on AgentCore](https://explainx.ai/blog/trovata-treasury-agents-bedrock-agentcore-aws-2026)** — UAEF is how a platform team regression-tests those graphs across releases. - **[American Express workforce agent rollout](https://explainx.ai/blog/american-express-5-percent-workforce-cuts-ai-agents-mcp-2026)** — enterprise scale is where Dunbar-number reporting problems appear; UAEF is the technical mirror of "what broke after the model swap."