{"slug": "automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions", "title": "Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions", "summary": "Amazon Web Services (AWS) published a guide on building a CI/CD quality gate for AI agents using Amazon Bedrock AgentCore and GitHub Actions, which deploys an agent with role-based MCP tools, evaluates it via the AgentCore Evaluate API, and blocks pull requests when evaluation scores drop below a threshold such as 0.8 out of 1.0. The reference implementation is available in the awslabs/agentcore-samples repository.", "body_md": "## [Artificial Intelligence](/blogs/machine-learning/)\n\n# Automated agent evaluation with Amazon Bedrock AgentCore and GitHub Actions\n\n*Build a continuous integration and continuous delivery (CI/CD) quality gate that deploys an agent with role-based MCP tools, evaluates it, and blocks PRs when evaluation scores drop.*\n\nYou shipped an AI agent on [Amazon Bedrock AgentCore runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is.html). It calls tools through an [MCP server](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-mcp.html) protected by OAuth. Now you want CI to tell you when a code change makes its performance worse before it reaches production.\n\nThis post walks through a GitHub Actions pipeline that deploys an agent to AgentCore runtime and evaluates the agent with evaluation prompts using the [AgentCore Evaluate API](https://docs.aws.amazon.com/bedrock-agentcore/latest/APIReference/API_Evaluate.html). If the agent regresses, the PR fails.\n\nWe’ll cover the full stack: a [Strands](https://strandsagents.com/) agent that connects to an MCP server with role-based access control, a shared Cognito pool serving both machine-to-machine (M2M) and user-scoped auth flows, CDK infrastructure-as-code, and a unified evaluation script. The complete reference implementation is available in the [accompanying repository](https://github.com/awslabs/agentcore-samples/tree/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation).\n\n## What this covers\n\n- Deploying an agent + MCP server to AgentCore runtime using CDK.\n- Role-based access control on MCP tools (three-layer auth pattern).\n- Invoking OAuth-protected runtimes from CI (M2M `client_credentials` flow).\n- Running on-demand evaluations with built-in evaluators.\n- Enforcing a quality gate that blocks merges on regression.\n- Handling the OAuth challenge: how CI pipelines authenticate without user context.\n\n## Key concepts\n\nBefore diving in, here’s a quick primer on the building blocks. Skip ahead if you’re already familiar with them.\n\n- [**AgentCore runtime**](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime.html) is a managed hosting platform for AI agents. You deploy your agent code (Python, any framework), and AgentCore handles scaling, session isolation, and infrastructure. Think of it as AWS Lambda for agents.\n- [**AgentCore Evaluations**](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations-types.html) , a capability of Amazon Bedrock AgentCore, scores agent behavior using a large language model (LLM) as a judge. It reads[OpenTelemetry](https://opentelemetry.io/) traces from Amazon CloudWatch and rates responses on dimensions like helpfulness, correctness, and tool selection accuracy.\n- [**MCP (Model Context Protocol)**](https://modelcontextprotocol.io/) is an open protocol that agents use to call external tools through a standardized interface. An MCP server exposes tools. The agent discovers and calls them. AgentCore runtime can[host MCP servers](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-mcp.html) and connect agents to them.\n- [**OpenID Connect (OIDC) federation**](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services) is how GitHub Actions assumes an AWS Identity and Access Management (AWS IAM) role without storing long-lived credentials. GitHub issues a short-lived token, AWS validates it, and the workflow gets temporary credentials.\n- **Quality Gate** is a CI/CD pattern where a pipeline step must pass a threshold before the build can proceed. In our case, the agent’s evaluation scores must meet a minimum bar (for example, 0.8 out of 1.0) or the PR stays blocked.\n\n**Why this matters:** Without automated evaluation, agent quality is subjective. A developer changes a system prompt. The agent starts giving worse answers, and nobody notices until users complain. A quality gate catches this at PR time before it reaches production.\n\n## The problem\n\nHere’s the scenario. You have an agent deployed on AgentCore runtime. It calls tools through an MCP server where some tools are public. Others are restricted by user role. Every time someone changes the system prompt, swaps a model, or updates tool configurations, you want to know: did the agent get better or worse?\n\nManual testing doesn’t scale. You need automated evaluation in CI. That means automatically deploying the agent in a dev environment, invoking it with representative prompts, scoring the responses, and blocking the merge if quality drops.\n\nThe complication: your MCP server uses OAuth with role-based access control. CI pipelines don’t have user context. How do you authenticate a headless pipeline against an OAuth-protected agent that forwards tokens to an MCP server expecting user roles?\n\n## AgentCore Evaluations: Where it fits\n\n[AgentCore Evaluations](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html) is the quality measurement layer in the Amazon Bedrock AgentCore platform. It sits alongside [AgentCore runtime](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime.html), which hosts your agent, and [AgentCore Observability](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/observability-get-started.html), a capability of Amazon Bedrock AgentCore that captures traces, completing the build → deploy → observe → evaluate lifecycle.\n\nThe service scores agent interactions using LLM-as-a-judge by default, with an option for code-based evaluation via AWS Lambda. It operates on OpenTelemetry traces, the same traces your agent already emits through AgentCore Observability. For on-demand evaluation, you provide span data directly in the API call; online and batch evaluation read from CloudWatch.\n\n**Three evaluation modes** cover different stages:\n\n- **On-demand evaluation** evaluates specific sessions at any time. You provide span data, pick your evaluators, and get scores back. This is what powers CI/CD quality gates, the focus of this post.\n- **Online evaluation** continuously monitors production traffic with configurable sampling rates. Results feed into CloudWatch dashboards for trend monitoring.\n- **Batch evaluation** scores multiple sessions in a single asynchronous job. You point it to your CloudWatch Logs, pick your evaluators, and get aggregate plus per-session results. This is what powers baseline measurement and pre/post regression testing.\n\nEvaluators come in four categories:\n\n- Built-in evaluators cover common quality dimensions: Helpfulness, Correctness, GoalSuccessRate, ToolSelectionAccuracy, ToolParameterAccuracy, and more. They operate at session, trace, and tool-call levels. Three trajectory evaluators (`TrajectoryExactOrderMatch` ,`TrajectoryInOrderMatch` ,`TrajectoryAnyOrderMatch` ) compare actual tool-call sequences against expected trajectories.\n- Custom evaluators use your own LLM-as-a-judge prompts for domain-specific scoring. Ground truth fields (expectedResponse, assertions, expectedTrajectory) are available as placeholders in custom evaluator prompts too.\n- Code-based evaluators run a Lambda function against each trace or session and return a score, label, and explanation calculated by your custom implementation. Use them for deterministic checks like regex matching, schema validation, or keyword presence without LLM costs.\n- Third-party evaluators from the DeepEval and AutoEval open-source libraries are managed by the service like built-in evaluators. Select one by ID with no model or configuration required. You can also derive a custom evaluator from a built-in or third-party evaluator to run its logic on your own model.\n\nThe Evaluate API accepts `sessionSpans` (OpenTelemetry trace data from CloudWatch) and returns structured scores. Each `evaluate()` call must contain spans from a single session only. Mixing sessions causes a `ValidationException`.\n\nThe API also accepts optional ground truth through `evaluationReferenceInputs`. You can provide an `expectedResponse` (used by Correctness), `assertions` (used by GoalSuccessRate), or an `expectedTrajectory` (used by trajectory evaluators). Traces without ground truth fall back to ground-truth-free evaluation, so you only need to provide it for the turns you care about.\n\n## Architecture\n\nThe pipeline deploys two AgentCore runtimes behind a shared Cognito user pool. One AgentCore runtime for the Strands agent and one for the MCP server:\n\nA single Cognito user pool serves two auth flows:\n\n| **Flow** | **Grant Type** | **Token Content** | **MCP Tool Access** | \n| M2M (CI pipelines) | `client_credentials` | Scopes only | All tools (no role check) | \n| User (interactive) | `authorization_code` | Scopes + `custom:roles` | Role-gated tools enforced | \n\nThe GitHub Actions pipeline deploys the agent stack to a dev environment, retrieves a JWT token from the provisioned Cognito instance to authenticate API calls, invokes the agent with an evaluation dataset, and analyzes the generated traces in Amazon CloudWatch Logs to assess performance against defined thresholds. It automatically approves or blocks the PR based on whether the overall score meets the acceptance criteria.\n\n## Handling OAuth-protected MCP servers in CI\n\nWhen your agent calls an MCP server protected by OAuth, CI pipelines face a challenge: they don’t have user context. The MCP server expects a JWT with role claims, but a headless CI runner can’t complete an interactive OAuth consent flow.\n\nThere are three approaches for evaluating the agent, each with different trade-offs:\n\n### Approach A: Evaluate stored traces\n\nDecouple evaluation from live MCP calls entirely. A staging pipeline runs the agent with representative prompts, captures traces, and commits them as JSON fixtures. PR-time, CI evaluates those stored traces. No live invocation is needed.\n\nThe Evaluate API doesn’t need a live agent. It scores OpenTelemetry spans you provide. Your CI pipeline becomes deterministic (check existing traces) and you sidestep the OAuth problem completely.\n\n**Trade-off:** You’re evaluating the staging deployment’s behavior, not the code in the current PR. The accompanying repo includes [scripts/evaluate_stored_traces.py](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/scripts/evaluate_stored_traces.py) and sample fixtures in `fixtures/` to get started with this approach.\n\n### Approach B: Service account with pre-authorized consent\n\nCreate a dedicated test user in your identity provider. Complete the OAuth consent flow once (interactively), cache the refresh token in [AWS Secrets Manager](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html). CI uses this token to invoke the agent as that test user.\n\n**Trade-off:** Refresh tokens expire. You need a rotation mechanism or periodic manual re-consent.\n\n### Approach C: M2M auth (this post’s approach)\n\nConfigure your MCP servers to support both M2M and user-scoped grant types. CI uses M2M tokens while interactive users go through the standard OAuth consent flow.\n\nThe MCP server middleware distinguishes between token types: M2M tokens contain scopes but no roles, so role checks are bypassed, and all tools are accessible. User tokens carry `custom:roles` claims, so tool-level access control is enforced. This bypass is secure because M2M tokens require a client secret that’s never exposed to end users. Only CI pipelines and the agent runtime can obtain these tokens, preventing untrusted callers from acquiring role-less tokens.\n\n**Trade-off:** M2M tokens bypass role checks by design. If you need CI to test role enforcement specifically, use Approach B.\n\nUse the decision tree below to find out which approach suits your use case:\n\n|  | **Approach A** | **Approach B** | **Approach C** | \n| Live invocation? | No | Yes | Yes | \n| MCP compatibility | All servers | All servers | Requires dual-token auth support | \n| CI determinism | High | Medium | Medium | \n| Role testing? | No | Yes | No (M2M bypasses roles) | \n| Best for | Quick start | Full E2E with roles | Internal tool agents | \n\n**Tip:** Start with Approach A to get a quality gate running quickly. Graduate to Approach C (this post) for full end-to-end CI that tests the actual PR’s code changes.\n\n## Prerequisites\n\n- AWS account with [AgentCore](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/what-is-bedrock-agentcore.html) access and[CDK bootstrapped](https://docs.aws.amazon.com/cdk/v2/guide/bootstrapping.html) .\n- Docker installed and running.\n- Python 3.12+, Node.js 20+.\n- Install the required Python packages: `pip install boto3 requests bedrock-agentcore-starter-toolkit`\n\n**Note:** The Evaluation class from bedrock-agentcore-starter-toolkit handles trace collection from CloudWatch and scoring automatically, so you don’t need to manually query log groups or call the raw Evaluate API.\n\n## MCP server: three-layer auth\n\nFor Approach C, the MCP server uses three layers to support both M2M and user-scoped tokens. This is the key pattern that makes CI evaluation work alongside production role enforcement.\n\n**Layer 1 JWT validation (AgentCore):** The platform validates signature, issuer, audience, and expiry before the request reaches your code. No implementation needed. AgentCore handles this through the [Custom JWT Authorizer](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/inbound-jwt-authorizer.html).\n\n**Layer 2 Header passthrough:** `request_header_allowlist=[\"Authorization\"]` on both runtimes makes sure the JWT reaches the agent and MCP containers. AgentCore forwards the caller’s Authorization header to your container unchanged.\n\n**Layer 3 Role-based tool access (AuthMiddleware):** A FastMCP native middleware that reads the JWT through `fastmcp.server.dependencies.get_http_headers()`, decodes claims using PyJWT, and enforces `custom:roles` against the tool `meta`. M2M tokens (scopes but no roles) get full access. User tokens need the right role.\n\nThe middleware is added directly to the FastMCP server instance:\n\n## Infrastructure: CDK stack\n\nThe CDK stack deploys everything in one command: Cognito pool, both runtimes, IAM roles, and pre-created test users. See [infrastructure/stack.py](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/infrastructure/stack.py) for the full implementation.\n\nKey resources created by the stack include:\n\n- A Cognito domain.\n- An M2M app client (`client_credentials` flow) for CI.\n- A user app client (`authorization_code` flow) for interactive use.\n- Two pre-created users: `user-a` (FinanceUser) and`user-b` (HRUser).\n- An MCP server AgentCore runtime (protocol: MCP) with JWT authorizer.\n- A Strands agent AgentCore runtime (protocol: HTTP) with JWT authorizer.\n\nThe GitHub Actions workflow deploys the CDK stack in a dev environment and uses the created resources to run evaluation.\n\n## Evaluation script\n\nThe unified evaluation script ([scripts/agentcore_eval.py](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/scripts/agentcore_eval.py)) handles the full pipeline: get token, wait for runtime, invoke agent, wait for traces, run evaluations, and gate on threshold.\n\nThe token acquisition uses the standard `client_credentials` grant.\n\nAgent invocation uses HTTPS with a Bearer token (not boto3):\n\nThe script uses bedrock-agentcore-starter-toolkit’s Evaluation class to run evaluations, which handles trace collection from CloudWatch automatically:\n\nEvaluation prompts cover the agent’s full tool surface including built-in tools, public MCP tools, and role-gated MCP tools:\n\n## GitHub Actions workflow\n\nThe workflow runs on every PR to main that touches agent code, MCP server, infrastructure, or scripts. It deploys the CDK stack, invokes the agent, runs evaluations, posts results as a PR comment, and tears down the stack. See the [full workflow](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/.github/workflows/agentcore-eval.yml) for the complete implementation.\n\nThe key steps are shown below:\n\n**Warning:** Runtimes stay in CREATING for a few minutes after CDK deploy returns, and invoking one before it’s READY fails with 424 Failed Dependency. The workflow polls `get_agent_runtime` (via the `bedrock-agentcore-control` client) until both are READY, then warms up the MCP server before evaluating.\n\n## CI/CD setup\n\nConfigure the following components to activate automated agent evaluation in your CI/CD pipeline.\n\n### 1. Create GitHub OIDC provider (one-time)\n\n### 2. Create IAM role for GitHub Actions\n\nCreate a role with trust policy for your repo and permissions for CDK, Amazon Bedrock AgentCore, Amazon Cognito, Amazon Elastic Container Registry (Amazon ECR), and Amazon Bedrock. See the [accompanying repo README](https://github.com/awslabs/agentcore-samples/tree/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation#cicd-setup) for the full policy.\n\n### 3. Add GitHub secret\n\n| **Secret** | **Value** | \n| AWS_ROLE_ARN | ARN of the IAM role above | \n\nEverything else is read from CDK outputs at runtime.\n\n## Built-in evaluators reference\n\nAgentCore provides [multiple built-in evaluators](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/prompt-templates-builtin.html) organized by what they assess:\n\n| **Category** | **Evaluators** | **When to Use** | \n| SESSION | GoalSuccessRate | Did the conversation achieve the user’s goals? | \n| TRACE | Helpfulness, Correctness, Coherence, Conciseness, Faithfulness, InstructionFollowing, ResponseRelevance, ContextRelevance, Harmfulness, Refusal, Stereotyping | Per-request quality and safety | \n| TOOL_CALL | ToolSelectionAccuracy, ToolParameterAccuracy, TrajectoryExactOrderMatch, TrajectoryInOrderMatch, TrajectoryAnyOrderMatch, SkillSelectionAccuracy, SkillInstructionFollowing | Was the right tool called with the right parameters? Trajectory evaluators check tool-call order (requires expectedTrajectory ground truth). Skill evaluators check skill selection and instruction adherence. | \n\nThis post uses four evaluators: GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy. The tool-call evaluators are particularly relevant for agents with MCP tools. They verify the agent picks the right tool and passes correct parameters.\n\n**Tip:** Start with four to five evaluators for CI. Add trajectory evaluators for tool-calling agents and safety evaluators (Harmfulness, Stereotyping, Refusal) for customer-facing agents. Use code-based evaluators for deterministic checks. Use the full set of periodic deep evaluations.\n\n## Testing the pipeline: Fail, fix, pass\n\nA reliable way to build confidence in a quality gate is to watch it catch a real regression.\n\n**Deliberate failure:** Change the agent’s system prompt to something unhelpful:\n\nPush to a feature branch and open a PR. The pipeline runs and you’ll see:\n\n**Note:** ToolSelectionAccuracy might still pass because the agent may select the right tool even with a bad system prompt. The evaluators measure different dimensions independently.\n\n**Fix and pass:** Restore a proper system prompt, push again. The pipeline re-runs:\n\nThe PR is unblocked.\n\n**Warning:** `LLM-as-judge` scores have inherent variance. As a result, the same prompt evaluated twice may produce slightly different scores. Set your threshold with some margin below your target reliability to account for this variance.\n\n## Lessons from testing\n\nWe ran this pipeline end-to-end. Here are the gotchas. Save yourself the debugging time.\n\n1. To invoke OAuth-protected AgentCore runtimes, POST directly to the HTTPS endpoint with a Bearer token rather than using the `invoke_agent_runtime()` method in`boto3` .\n2. **Trace propagation takes 30-90 seconds.** The evaluation script retries every 30 seconds for up to 10 minutes. Don’t query CloudWatch immediately after invocation.\n3. **ARM64 images required.** AgentCore runtime requires ARM64 containers. GitHub runners are x86_64. Therefore, use QEMU + Docker Buildx for cross-compilation.\n4. **Runtime restart after CDK deploy.** Invoking a runtime before it’s READY fails with 424 Failed Dependency, so the workflow polls`get_agent_runtime` until both runtimes are ready (and warms up the MCP server) before invoking.\n5. **`sessionSpans` is the API parameter name.** The Evaluate API accepts`evaluationInput: {\"sessionSpans\": [...]}` . Each call must contain spans from a single session only. Mixing sessions causes a`ValidationException` .\n6. **Timestamps must be integers.** OpenTelemetry nanosecond timestamps like`startTimeUnixNano` must be JSON integers, not quoted strings. String timestamps cause a`ValidationException` .\n\n## Adapting for Microsoft Entra ID\n\nThis post uses Amazon Cognito, but the architecture is identity-provider-agnostic. If your organization uses Microsoft Entra ID (formerly Azure AD), the same pipeline works with targeted changes:\n\n- **Token endpoint:** Change the regional endpoint of Amazon Cognito to`https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token` .\n- **Discovery URL:** Change to`https://login.microsoftonline.com/{tenant_id}/v2.0/.well-known/openid-configuration` .\n- **Client credentials:** Replace Cognito client ID and audience with Entra ID application ID and scope URI.\n\nEverything else stays the same. The deploy script’s `authorizerConfiguration.customJWTAuthorizer` structure is identical, the evaluation script doesn’t touch authentication (it uses IAM through `boto3`), and the GitHub Actions workflow structure is unchanged.\n\n## Trade-offs and limitations\n\n- **Pipeline takes ~10 minutes.** CDK deploy + runtime startup + trace propagation + evaluation. Fine for PR gates, too slow for pre-commit.\n- **`LLM-as-judge` has inherent variance.** The same trace evaluated twice may produce slightly different scores. Set thresholds with margin.\n- **Cost per run.** Each evaluator invocation calls the judge model. 4 evaluators × 5 prompts = 20 judge calls per PR. Monitor Bedrock costs at scale.\n- **M2M tokens bypass role checks.** By design, CI needs access to all tools. If you need CI to test role enforcement, use Approach B (service account).\n\n**Note:** Despite these limitations, automated evaluation is strictly better than no evaluation. Even imperfect quality gates catch obvious regressions that manual review misses.\n\n## Clean up\n\nThe accompanying repo provisions two AgentCore runtimes, a Cognito user pool, IAM roles, and pre-created users, so tear everything down when you’re finished to stop incurring cost. A single command removes it all:\n\nIf you deployed with npx rather than a global CDK CLI, run `npx aws-cdk@2 destroy --force` instead. In the dev stack, `cdk destroy` removes both runtimes, the Cognito pool and its app clients, the pre-created users, the IAM roles, and the M2M client secret in AWS Secrets Manager. The secret’s removal policy is set to destroy, so repeated deploys and teardowns stay clean. The CI workflow tears the same stack down automatically at the end of every run, because its CDK destroy step runs with `if: always()`. You only need this command for stacks you deploy yourself while following along.\n\n## Key takeaways\n\n1. **Three-layer MCP auth** (platform JWT validation to middleware claim extraction to tool-level role checks) cleanly separates concerns and supports both M2M and user-scoped flows without code changes.\n2. **CDK deploys everything.** One`cdk deploy` creates the Cognito pool, both runtimes, IAM roles, and pre-created users. One`cdk destroy` tears it all down.\n3. **The `bedrock-agentcore-starter-toolkit` simplifies evaluation.** The Evaluation class handles trace collection from CloudWatch and scoring, so there’s no need to manually query log groups and call the raw Evaluate API with`sessionSpans` .\n4. **Token forwarding bridges CI and production.** The agent inspects the incoming JWT to determine if it’s a user token (forward to MCP for role checks) or M2M token (use shared client, bypass roles). Same code serves both callers.\n5. **The Evaluate API is decoupled from the agent runtime.** You don’t need a running agent to evaluate traces. This is the key insight that makes Approach A (stored traces) work, and it’s a direct path to a CI quality gate.\n6. **Start with four evaluators and expand.** GoalSuccessRate, Correctness, ToolSelectionAccuracy, and ToolParameterAccuracy cover the basics. Add safety evaluators for customer-facing agents.\n7. Ground truth and code-based evaluators extend the toolkit. Trajectory evaluators are programmatic and offered at no additional cost, ideal for CI. Code-based evaluators run deterministic Lambda checks alongside `LLM-as-judge` scoring in the same evaluation call.\n\n## Try it yourself\n\nThe [accompanying repository](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/) includes the complete implementation: CDK infrastructure for Cognito and both runtimes, an agent with MCP client and token forwarding, an MCP server with role-based access control, a unified evaluation script, a GitHub Actions workflow, and two [walkthrough scripts](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/scripts/). Use `scripts/deploy_and_test_rbac.py` for deployment and role-based access testing, and `scripts/evaluation_pipeline.py` for the evaluation pipeline.\n\nTo extend the project, you can start with Approach A by running `python3 scripts/evaluate_stored_traces.py` against the bundled fixtures without any deployment. From there, try adding a new role-gated tool to the MCP server (see [mcp-server/README.md](https://github.com/awslabs/agentcore-samples/blob/main/01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/cicd-gated-evaluation/mcp-server/README.md)) or creating a custom evaluator with your own `LLM-as-judge` prompt. Maybe try adjusting `EVAL_THRESHOLD` per environment (for example, 0.7 for dev, 0.8 for staging, 0.9 for prod). You can also set up [online evaluation](https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/create-online-evaluations.html) for continuous production monitoring or compare on-demand to online evaluation for your use case.", "url": "https://wpnews.pro/news/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions", "canonical_source": "https://aws.amazon.com/blogs/machine-learning/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions/", "published_at": "2026-09-08 16:23:21+00:00", "updated_at": "2026-09-08 16:55:20.875905+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "developer-tools", "ai-infrastructure"], "entities": ["Amazon Web Services (AWS)", "Amazon Bedrock AgentCore", "GitHub Actions", "MCP", "AgentCore Evaluate API", "Strands", "CDK", "awslabs/agentcore-samples"], "alternates": {"html": "https://wpnews.pro/news/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions", "markdown": "https://wpnews.pro/news/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions.md", "text": "https://wpnews.pro/news/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions.txt", "jsonld": "https://wpnews.pro/news/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions.jsonld"}}