AWS UAEF: Universal Agent Evaluation Across 36 Metrics

AWS published the Universal Agent Evaluation Framework (UAEF) on November 11, 2026, an open, framework-agnostic evaluation layer that normalizes traces from LangGraph, LangChain, Strands, Bedrock agents, AgentCore, Langfuse, or custom JSON into a single AgentTrace and scores them against 36 built-in metrics across 7 dimensions. The code is available on GitHub as awslabs/uaef under Apache 2.0, with results persisted to DynamoDB and S3 and judged by deterministic metrics plus a Bedrock LLM-as-judge with prompt-injection guards. AWS said 12 customer organizations in telecom, healthcare, and financial services used UAEF before the public post, positioning it for release gating rather than demos.

On November 11, 2026, AWS published the Universal Agent Evaluation Framework UAEF https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/?ref=explainx — an open evaluation layer that answers a question every agent team hits after the demo: did the prompt, model, or tool change actually improve behavior? Instead of rebuilding metrics per SDK, UAEF adapters normalize LangGraph, LangChain, Strands, Bedrock agents, AgentCore, Langfuse, or custom JSON into one AgentTrace , then score the same 36 metrics whether the agent ran locally, behind HTTP, or on a managed runtime. The code lives on GitHub as awslabs/uaef https://github.com/awslabs/uaef?ref=explainx under Apache 2.0. AWS says 12 customer organizations in telecom, healthcare, and financial services used UAEF before the public post — a signal this is meant for release gating, not slideware. TL;DR | Piece | AWS Nov 11, 2026 | |---|---| | Project | UAEF — Universal Agent Evaluation Framework | | License | Apache 2.0 awslabs/uaef | | Adapters | LangGraph, LangChain, Bedrock, Langfuse, Strands , AgentCore , Generic JSON | | Metrics | 36 built-in across 7 dimensions | | Storage | DynamoDB metadata + S3 full JSON per experiment | | Judges | Deterministic metrics + Bedrock LLM-as-judge with prompt-injection guards | | Surfaces | Python library, notebooks, deployable web UI , optional EKS API | The problem UAEF targets Teams on AWS already mix Strands decider patterns https://explainx.ai/blog/amazon-strands-decider-internal-ticketing-sonnet-replacement-2026 , Bedrock AgentCore https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026 , and LangGraph graphs in different products. Each framework logs messages, tool calls, and handoffs differently, so evaluation code rots per stack — exactly the fragmentation WorkflowArena-style orchestration benchmarks https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026 try to measure from the outside. UAEF separates normalization from scoring : adapters emit AgentTrace; every metric consumes that shape. AWS claims you can compare an agent built on one framework against another on one scale — useful when a platform team standardizes gates but not SDK choices. Closed-loop flow AWS diagrams a repeatable loop: 1. Adapter ingest — pass an existing trace or let UAEF invoke the agent via ARN/HTTP and capture fresh output. 2. PII screen — detect emails, phone numbers, payment data before storage. 3. Evaluate — run selected metrics subset or full 36 . 4. Persist — optional experiment row in DynamoDB plus full results in S3. 5. Compare & report — highlight regressions vs baseline; optional LLM-written insights on what to change next. That maps directly to CI patterns explainx.ai recommends in Raindrop reasoning memory benchmarks https://explainx.ai/blog/raindrop-reasoning-agent-memory-tau2-bench-67-percent-2026 and Infrathon infrastructure agent evals https://explainx.ai/blog/infrathon-long-horizon-ai-infrastructure-agent-benchmark-2026 — except UAEF is AWS-native storage with a shipped UI. Metrics and judges Seven dimensions span behaviors a single accuracy number hides: | Dimension | What it catches | |---|---| | Tool calling | Wrong tool, bad argument shape, sequence errors | | Response quality | Relevance, completeness, hallucination vs context | | Responsible AI | Safety, bias, toxicity, prompt injection resistance | | Performance | Latency, token and cost efficiency | | Multi-turn | Retention, coherence, goal completion | | Multi-agent | Delegation, utilization, workflow completion | | Reasoning | Chain consistency, fallacies | Deterministic metrics replay identically; judge metrics call Bedrock with untrusted-output handling — rubric separated from content, escaped delimiters, evidence spans required or scores reject. AWS explicitly says judge scores are not a safety gate; use Bedrock Guardrails https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails.html?ref=explainx on the runtime path for enforcement — same split we document in our agent sandbox security post https://explainx.ai/blog/claude-cage-vm-sandbox-claude-code-network-isolation-2026 . Integrations include RAGAS and DeepEval metrics for retrieval-heavy agents. Experiment tracking builders actually need When persist=True , UAEF records framework, model, tools, prompts, optional git SHA, and per-metric averages. Mark a baseline run and the comparison engine flags regressions — the workflow MemCurrent-style memory papers chase with token counts, but for full agent traces . Multi-tenant note: AWS's SECURITY.md says UAEF expects host-level tenant isolation — separate stacks or accounts per customer — and scopes reads to Cognito sub inside a deployment. Do not bolt multi-customer SaaS onto one shared UAEF stack without that separation. How to start this week 1. git clone https://github.com/awslabs/uaef and uv sync per README. 2. Run notebook 00 langgraph.ipynb or 06 agentcore.ipynb against a toy agent. 3. Pick metrics from AWS's scenario table RAG Q&A vs tool-using vs multi-turn columns in the blog . 4. Call batch evaluate ..., persist=True, experiment name='v2 prompt' before merging prompt changes. 5. Optional: deploy uaef-service with CDK when you want a reviewer UI on top of the same engine. AWS also documents dataset columns — Question , optional Answer , expected tool calls JSON, and session id / turn id for conversations — so exports from Quick Skills agent catalogs https://explainx.ai/blog/aws-quick-skills-agents-catalog-build-guide-2026 or internal ticket bots can reuse CSVs with minimal reformatting. Playbook highlights from the AWS post AWS's metric chooser table is the fastest onboarding doc: start from scenario , not the full metric list. Retrieval agents begin with faithfulness and context precision; tool agents need selection accuracy and parameter quality; compliance-heavy flows prioritize safety and injection detection. Pipeline choices are explicit: passive trace evaluation vs active invoke helpers, single trace vs batch, offline CI vs OnlineEvaluator on live traffic, ephemeral scores vs persisted experiments only persisted runs compare . Deployment surfaces in more detail AWS documents three runnable surfaces that share one evaluation core: Python library and notebooks. The fastest path for a platform team is evaluate on a trace captured from staging, then batch evaluate on a CSV exported from support tickets. Notebooks under notebooks/ map frameworks to scenarios — LangGraph batch runs, AgentCore multi-agent sessions, Langfuse imports for production traces you already store. Serverless web UI. The uaef-service CDK stack provisions Cognito auth, API Gateway, worker Lambdas, Step Functions, DynamoDB, S3, and CloudFront for a React reviewer UI. AWS stresses that cors allowed origins is mandatory: blank means the deploy script derives the CloudFront origin on first UI deploy, while EKS mode requires an explicit origin because no distribution exists. EKS API. Teams that already run Bedrock inference on Kubernetes may prefer the containerized FastAPI deployment in deploy eks/ — same persistence schema, different ingress. Bring your own SSO and UI; keep experiment comparison on the Python API. Across modes, only persisted experiments compare . Ephemeral evaluate calls help debug one bad trace; they will not show up in regression dashboards. Example metric bundles AWS recommends Rather than enabling all 36 metrics on day one, AWS's scenario table implies starter packs: | Agent shape | Start with | Why | |---|---|---| | RAG helpdesk | answer relevance , hallucination score , ragas faithfulness | Catches wrong retrieval and invention without tool metrics | | Ticket router | tool selection accuracy , parameter quality , latency score | Measures wrong API and slow escalations | | Compliance bot | safety score , prompt injection detection , toxicity score | Separates policy incidents from helpfulness | | Pair programmer | accuracy , completeness , reasoning chain metrics | Aligns with IDE agent regressions | | Swarm workflow | delegation quality , workflow completion , coordination efficiency | Mirrors WorkflowArena https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026 failure modes | Add multi-turn metrics only when datasets include session id and turn id columns; otherwise UAEF treats rows as independent single-turn cases. CI/CD pattern A minimal release gate looks like this: 1. Export 50–200 production queries with redacted PII into a CSV using PIIDetector . 2. Mark the current production agent run as baseline in UAEF experiment metadata. 3. On each pull request that changes prompts, tools, or model ARNs, run batch evaluate ..., persist=True, experiment name='pr-1234' . 4. Fail the build if any dimension drops more than an agreed tolerance versus baseline averages. 5. Attach the HTML report link to the PR for human review of judge-reasoning fields stored in S3. This is the AWS-native version of patterns Amazon's Bedrock evaluation playbook https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026 describes for DeepEval inside AgentCore — complementary, not redundant. Judge security details worth copying AWS's judge path treats agent output as untrusted input : rubric and content are separated, content is escaped, and judge answers must cite verbatim evidence spans or the score is rejected. That design responds to the same class of attacks Claude memory exfiltration research https://explainx.ai/blog/claude-memory-heist-web-fetch-exfiltration-ayush-paul-july-2026 documents — models that see their own evaluation prompt can game it. UAEF explicitly says do not use judge scores as runtime policy. Pair UAEF offline gates with Bedrock Guardrails or application-level tool allowlists on the live agent. Online evaluation versus offline batches AWS distinguishes OfflineEvaluator batch runs on frozen datasets from OnlineEvaluator hooks that score live conversations turn by turn. Offline runs belong in CI and release trains; online hooks belong in canary deployments where you sample a percentage of production traffic with PII redaction to catch drift when user phrasing shifts. The blog's passive versus active pipeline choice matters here: passive mode ingests traces your observability stack already captured Langfuse, CloudWatch logs, custom JSON ; active mode lets UAEF invoke agents so every test uses the same temperature and tool manifest. Mixed environments often passive-import staging traces but actively invoke production builds before promotion. Human-in-the-loop review UAEF documentation references HITL flows for disputed judge scores — important when LLM judges disagree with deterministic metrics on borderline safety cases. Treat HITL queues as part of the product, not an afterthought: without them, teams mute judge metrics and ship anyway. When not to adopt UAEF yet UAEF is heavy if you have one LangGraph demo and no baseline traffic. It pays off when: - Multiple teams ship agents on different frameworks under one platform org. - You already pay Bedrock judge costs and need shared storage. - Regulated customers ask for experiment audit trails . If you only need a single RAG faithfulness number, starting with RAGAS inside the app may be faster — then graduate to UAEF when regressions become cross-team. How UAEF fits next to other AWS agent posts - Bedrock tokenomics data lake https://explainx.ai/blog/amazon-bedrock-tokenomics-data-lake-finops-2026 — UAEF adds quality time series beside cost telemetry. - Trovata treasury agents on AgentCore https://explainx.ai/blog/trovata-treasury-agents-bedrock-agentcore-aws-2026 — UAEF is how a platform team regression-tests those graphs across releases. - American Express workforce agent rollout https://explainx.ai/blog/american-express-5-percent-workforce-cuts-ai-agents-mcp-2026 — enterprise scale is where Dunbar-number reporting problems appear; UAEF is the technical mirror of "what broke after the model swap."