# Amazon Bedrock AgentCore Evaluation Playbook: Built-In, DeepEval, and AutoEval

> Source: <https://explainx.ai/blog/amazon-bedrock-agentcore-evaluation-playbook-deepeval-autoeval-guide-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog>
> Published: 2026-11-10 00:00:00+00:00

**On November 10, 2026 AWS published an [agent evaluation playbook](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/?ref=explainx) for [Amazon Bedrock AgentCore Evaluations](https://aws.amazon.com/bedrock/agentcore/?ref=explainx)** — a practical map from **business outcomes** to **built-in**, **DeepEval**, and **AutoEval** judges. The post complements the [decision-model screening guide](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluations-decision-models-jev-strands-decider-screening-layer-guide-2026) AWS shipped six days earlier and the [seven-layer agent framework](https://explainx.ai/blog/aws-pluggable-agentic-ai-framework-bedrock-agentcore-seven-layers-2026) from November 9.

If you run agents on AgentCore — like the [Trovata treasury pattern](https://explainx.ai/blog/trovata-treasury-agents-bedrock-agentcore-aws-2026) AWS documented the same morning — this playbook answers a question decision models do not: **which full evaluators to run** after you know what "working" means.

## TL;DR

| Step | AWS recommendation | 
|---|---|
| 1 | Define success in **business terms** (first-contact resolution, sourced research, compliant answers). | 
| 2 | Split the outcome into **observable behaviors** across turns and tools. | 
| 3 | Map each behavior to an evaluator at **trace** or**session** level. | 
| 4 | Start with **two or three** evaluators; add more when a failure mode slips through. | 
| 5 | Use **on-demand** ,**online** , or**batch** mode with the**same evaluator IDs** . | 

## Why surface metrics hide agent failure

AWS opens with a warning teams already feel in production: high **completion rate** can coexist with **PII leakage**, skipped policy steps, or incoherent multi-turn drift. Latency and spot checks measure whether the agent **ran**, not whether it **met the business bar**.

Agent evaluation playbook diagram showing surface latency signals versus hidden agent behaviors on Amazon Bedrock AgentCore
*Image: [AWS](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/?ref=explainx)*

That framing matches what explainx.ai saw in [Raindrop's tau2-bench agent memory work](https://explainx.ai/blog/raindrop-reasoning-agent-memory-tau2-bench-67-percent-2026) and in [WorkflowArena orchestration benchmarks](https://explainx.ai/blog/workflowarena-multi-agent-orchestration-benchmark-einsia-2026): orchestration scores can look fine while **per-turn policy** fails. AgentCore Evaluations is AWS's hosted answer for measuring **below the waterline** behaviors at scale.

## Start with outcomes, not metric labels

The post's core discipline is simple and easy to skip:

1. **Business outcome** — e.g. first-contact resolution for support, fully sourced answers for research, zero false negatives for moderation.
2. **Observable behaviors** — goal completion, tool choice, compliance boundaries, skill steps.
3. **Evaluator mapping** — trace-level for per-turn signals; session-level for whole-conversation results.

Three-step flow from business outcomes to evaluators in the Bedrock AgentCore evaluation playbook
*Image: [AWS](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/?ref=explainx)*

AWS argues that picking "helpfulness" because the name sounds good produces numbers without insight. The same lesson applies outside Bedrock: [Andy Hall's audit-agent study](https://explainx.ai/blog/andy-hall-ai-audit-agents-false-alarms-malt-r-judge-2026) found high recall with costly false alarms when judges are not tied to explicit behaviors.

## Three evaluator tiers

AgentCore exposes three sources AWS diagrams as a control-versus-QA tradeoff:

| Tier | What you get | When AWS suggests it | 
|---|---|---|
| **Built-in** | AWS-authored IDs ( `Builtin.*` ) with AWS quality testing | Default starting point for common dimensions | 
| **Third-party managed** | DeepEval and AutoEval hosted like built-ins | When your dev stack already uses one library | 
| **Custom derived** | Swap judge model or rubric on a base evaluator | When catalog metrics almost fit | 

AgentCore evaluator catalog spanning built-in AWS, DeepEval, AutoEval, and custom judges
*Image: [AWS](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/?ref=explainx)*

### Built-in catalog (selected IDs)

AWS groups built-ins into:

- **Response quality:**`Builtin.Correctness` ,`Builtin.Faithfulness` ,`Builtin.Helpfulness` ,`Builtin.ResponseRelevance` ,`Builtin.Conciseness` ,`Builtin.Coherence` ,`Builtin.InstructionFollowing` ,`Builtin.Refusal` .
- **Task completion:**`Builtin.GoalSuccessRate` (session-level anchor for many stacks).
- **Tool use:**`Builtin.ToolSelectionAccuracy` ,`Builtin.ToolParameterAccuracy` .
- **Skills:**`Builtin.SkillSelectionAccuracy` ,`Builtin.SkillInstructionFollowing` .
- **Safety:**`Builtin.Harmfulness` ,`Builtin.Stereotyping` .
- **Trajectories:**`Builtin.TrajectoryExactOrderMatch` ,`Builtin.TrajectoryInOrderMatch` ,`Builtin.TrajectoryAnyOrderMatch` — require expected paths (batch / golden sets).

Exact rubrics live in the [AgentCore prompt template library](https://docs.aws.amazon.com/bedrock/latest/userguide/agentcore-evaluations.html?ref=explainx) per AWS. Call **`ListEvaluators`** to enumerate built-ins plus account customs.

### DeepEval managed metrics

DeepEval's managed set targets **multi-turn** failure modes AWS highlights:

- **Task and goal:**`ThirdParty.DeepEval.TaskCompletion` ,`ThirdParty.DeepEval.GoalAccuracy` ,`ThirdParty.DeepEval.ConversationCompleteness` .
- **Conversation quality:**`ThirdParty.DeepEval.TurnRelevancy` ,`ThirdParty.DeepEval.KnowledgeRetention` ,`ThirdParty.DeepEval.Summarization` .
- **Tool use:**`ThirdParty.DeepEval.ToolUse` .
- **Safety:**`ThirdParty.DeepEval.Bias` ,`ThirdParty.DeepEval.Toxicity` ,`ThirdParty.DeepEval.PIILeakage` .

`ConversationCompleteness` catches agents that answer the latest message but drop earlier user requests — a pattern we also track in [ChatGPT Business Agents hub launches](https://explainx.ai/blog/chatgpt-business-agents-business-hub-verified-brands-november-2026) where long-running tasks need session memory.

### AutoEval specialized signals

AutoEval adds narrower judges:

- **`ThirdParty.AutoEval.Security`** — instruction injection or system-prompt reveal attempts.
- **`ThirdParty.AutoEval.Possible`** — valid tasks wrongly declared impossible (over-refusal).
- **`ThirdParty.AutoEval.Humor`** — tone for consumer-facing agents.

Pair `AutoEval.Security` with governance patterns from [Okta cross-app access on AgentCore Gateway](https://explainx.ai/blog/okta-cross-app-access-bedrock-agentcore-gateway-cedar-policy-agent-governance-part-2-2026) when agents touch multiple SaaS tools.

## Multi-dimension evaluators

When **accuracy**, **policy compliance**, and **tone** can fail independently, AWS recommends a **multi-dimension custom LLM judge**: one trace read, **separate scores per dimension**, no blended composite that hides which criterion broke.

Keep criteria in **separate catalog evaluators** when:

- A built-in already covers them (`Builtin.Correctness` ,`ThirdParty.DeepEval.PIILeakage` ).
- They need **different levels** (trace vs session) or**different judge models** .
- Some need **ground truth** fields like`{expected_response}` — those are**batch-only** on live traffic.

This is the hosted version of a pattern [Hugging Face Decision Models](https://explainx.ai/blog/hugging-face-decision-models-systemone-api-2026) pursues for cheap per-turn gates — full judges stay expensive, so dimension design matters.

## Example stacks AWS publishes

AWS gives three starter tables. All mark trajectory and some correctness evaluators as **needing ground truth** (batch / golden data).

### Customer support

| Role | Evaluator | 
|---|---|
| Anchor | `Builtin.GoalSuccessRate` | 
| Start with | `ThirdParty.DeepEval.ConversationCompleteness` ,`ThirdParty.DeepEval.PIILeakage` | 
| Add when needed | `ThirdParty.DeepEval.Toxicity` ,`ThirdParty.AutoEval.Security` ,`Builtin.Helpfulness` ,`Builtin.SkillInstructionFollowing` | 

### Multi-step research

| Role | Evaluator | 
|---|---|
| Anchor | `Builtin.TrajectoryAnyOrderMatch` (with expected path) | 
| Start with | `ThirdParty.DeepEval.GoalAccuracy` ,`ThirdParty.DeepEval.KnowledgeRetention` | 
| Add when needed | `Builtin.Correctness` (with reference),`Builtin.Helpfulness` ,`Builtin.SkillInstructionFollowing` ,`ThirdParty.AutoEval.Possible` | 

### Regulated domain content

| Role | Evaluator | 
|---|---|
| Anchor | `Builtin.Correctness` (with reference) | 
| Start with | `ThirdParty.DeepEval.PIILeakage` ,`ThirdParty.AutoEval.Security` | 
| Add when needed | `ThirdParty.DeepEval.Bias` ,`Builtin.Helpfulness` | 

AWS notes **derived evaluators** for trajectory and skill metrics when **inference must stay in your account** — relevant for [Bedrock tokenomics and FinOps](https://explainx.ai/blog/amazon-bedrock-tokenomics-data-lake-finops-2026) teams metering judge spend.

## Three run modes (same IDs)

| Mode | Use case | AWS notes | 
|---|---|---|
| **On-demand** | Dev / debug via `client.evaluate()` | All evaluator types supported | 
| **Online** | Sample live AgentCore runtime traffic | Mix built-in and third-party in one config | 
| **Batch** | Pre-release regression on golden sessions | Ground-truth evaluators | 

The Python sample in the post uses `ThirdParty.DeepEval.TaskCompletion` with `sessionSpans` — the same ID works when you move from notebook to production online eval.

For physical-world agents, compare this software stack to [Crossnokaye Aura on AgentCore](https://explainx.ai/blog/crossnokaye-aura-atlas-bedrock-agentcore-physical-ai-2026): industrial agents need **trajectory** and **safety** dimensions, not helpfulness alone.

## How to adopt this without boiling the ocean

AWS's closing advice is deliberately minimal:

1. Answer **what success looks like** for this agent.
2. Name the **trace evidence** that proves it.
3. Pick **two or three** evaluators that target those behaviors.
4. Add evaluators when production or batch runs surface a **missed failure mode** .

If you already run [AgentCorruption-style hijack tests](https://explainx.ai/blog/agentcorruption-zenity-aws-bedrock-agentcore-hijack-every-agent-2026), treat `ThirdParty.AutoEval.Security` and PII judges as **continuous** checks, not pre-release one-offs.

Pair full judges with the [Jev / Strands Decider screening layer](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluations-decision-models-jev-strands-decider-screening-layer-guide-2026) when per-turn LLM judge cost dominates.

## What AWS does not publish here

The playbook is a **catalog and mental model**, not benchmark leaderboard numbers. AWS does not claim accuracy rates for each built-in judge in this post. Validation is on you: labeled edge cases, human majority vote, compare decision-model screeners to full judges — the same workflow AWS described in the November 4 decision-model article.

For agent **security incidents** rather than quality drift, see [november cyber tracker patterns](https://explainx.ai/blog/november-2026-cyberattacks-list-breaches-exploits-ai-run-attacks-tracker-2026) and evaluate whether your runtime needs action-level gates, not just transcript judges.

## Practitioner checklist before you turn on online eval

AWS assumes you already run agents on AgentCore with tracing enabled. Before you sample production traffic into evaluators, walk through this sequence — it mirrors the post's three-step model without requiring every catalog metric on day one.

**Define the failure you fear most.** For a coding agent, that might be merging unsafe shell commands. For a finance agent, leaking account numbers in summaries. Write that failure as a user-visible outcome, not as a metric name. If your team cannot articulate the outcome in one sentence, pause before buying judge capacity.

**Pick one session anchor and two trace checks.** Customer support stacks in the post anchor on `Builtin.GoalSuccessRate` and add conversation completeness plus PII leakage. Research stacks anchor on trajectory match when you have golden paths. Regulated stacks anchor on correctness with references. Your first week should not include every row in AWS's tables.

**Validate on labeled exports before online mode.** Export fifty to two hundred historical sessions with human labels on the behaviors you care about. Run on-demand `evaluate()` calls in a staging account. Compare built-in and DeepEval scores to human majority vote. Calibrate thresholds for online sampling only after agreement is acceptable on held-out sessions.

**Document judge model residency.** Managed third-party evaluators run on AWS capacity per the post. Custom derived evaluators exist so judge inference can stay inside your account when data governance requires it — the same reason [Amazon Quick Skills](https://explainx.ai/blog/aws-quick-skills-agents-catalog-build-guide-2026) teams segregate prompts and tools per tenant.

**Plan spend.** Each online eval multiplies inference cost on every sampled turn or session. Pair expensive judges with the [decision-model screeners](https://explainx.ai/blog/amazon-bedrock-agentcore-evaluations-decision-models-jev-strands-decider-screening-layer-guide-2026) AWS documented November 4 so only low-confidence spans hit a frontier judge.

**Re-run batch eval on every prompt or tool change.** AWS positions batch mode as your pre-release gate. Tie it to the same golden set you use for regression tests on [semantic caching](https://explainx.ai/blog/semantic-caching-llm-apps-cost-latency-guide-2026) or routing changes so eval debt does not drift from product debt.

## How this differs from generic LLM-as-judge tutorials

Community guides often show one helpfulness rubric and a Python script. The AgentCore playbook's value is **naming IDs** you can move from notebook to production without rewriting glue code, and **grouping** metrics so you do not double-pay for near-duplicate judges.

DeepEval's conversation metrics matter because enterprise agents are multi-turn by default now — [ChatGPT Business Agents](https://explainx.ai/blog/chatgpt-business-agents-business-hub-verified-brands-november-2026), [Claude office suite](https://explainx.ai/blog/anthropic-claude-meetings-whiteboards-sheets-office-suite-november-2026), and [Meta Muse wearables](https://explainx.ai/blog/meta-muse-apple-watch-wear-os-agent-2026) all assume sessions that stretch beyond a single prompt-response pair.

AutoEval's `Possible` metric addresses a failure mode safety teams see in red-team exercises: models that refuse valid work because caution prompts were overtuned. That is different from `Builtin.Refusal`, which measures policy declines on actually harmful requests.

If you are standardizing across clouds, compare this catalog to [Google's multimodal knowledge agent benchmark](https://explainx.ai/blog/multimodal-knowledge-agent-mka-benchmark-google-2026) work and to open harnesses like [Beneath the Surface agent evolution audit](https://explainx.ai/blog/beneath-the-surface-agent-harness-evolution-evoaudit-2026) — AgentCore is the managed AWS slice, not the whole eval universe.

## Related reading

Primary sources: [AWS evaluation playbook (Nov 10, 2026)](https://aws.amazon.com/blogs/machine-learning/the-agent-evaluation-playbook-built-in-deepeval-and-autoeval-on-agentcore/?ref=explainx) and [Amazon Bedrock AgentCore Evaluations documentation](https://docs.aws.amazon.com/bedrock/latest/userguide/agentcore-evaluations.html?ref=explainx).