# Gate AI Agent Deployments with Bedrock AgentCore and GitHub Actions > Source: > Published: 2026-09-25 19:10:06+00:00 Developers are shipping AI agents to production without tests. Not because they are reckless — because the tooling was not there. That changes now. AWS published a reference implementation that wires Amazon Bedrock AgentCore Evaluations into GitHub Actions, turning every pull request that touches an agent into a gated quality check. If your agent’s goal success rate drops, the merge is blocked. Same as your unit tests. It took the software industry a decade to build CI/CD culture around code; AI agents are about to get the same treatment. ## What AWS Shipped Amazon Bedrock AgentCore Evaluations went generally available in March 2026. AWS’s new reference implementation extends it into pull request workflows: when a PR modifies agent code, a system prompt, a model selection, or a tool configuration, GitHub Actions automatically deploys the agent to a dev environment, invokes it with a test dataset, retrieves the resulting [OpenTelemetry](https://opentelemetry.io/) traces from CloudWatch, scores them using AgentCore’s built-in evaluators, and blocks the merge if any score falls below a configured threshold. The full reference implementation is [available on GitHub](https://github.com/aws-samples/sample-bedrock-agentcore-runtime-cicd). ## The Four Evaluators That Gate Your PRs The reference implementation uses four evaluators from AgentCore’s catalog of 20+: - **GoalSuccessRate** — Did the agent actually complete the task the user gave it? - **Correctness** — Is the response accurate and appropriate given the prompt? - **ToolSelectionAccuracy** — Did the agent pick the right tool from its available set? - **ToolParameterAccuracy** — Did it derive the right parameters from context before calling the tool? This matters because most AI monitoring only checks the final output. These evaluators examine the reasoning path — whether the agent reached a good answer through a correct process or stumbled onto it by accident. The broader catalog also includes safety evaluators (Harmfulness, Stereotyping, Refusal) and trajectory checks for validating multi-step tool sequences. ## The Variance Problem Nobody Talks About Here is the implementation detail that will save you a frustrating afternoon: LLM-as-judge evaluation is non-deterministic. Run the same agent against the same prompts twice, and you might see scores of 0.84 and 0.79 with zero code changes. Set a hard threshold at 0.80 and you will generate random red builds that erode team confidence in the pipeline. The fix is straightforward: set thresholds with margin below your target reliability, and consider averaging across multiple evaluation runs before applying a pass/fail decision. AWS explicitly warns about this in their [implementation guide](https://aws.amazon.com/blogs/machine-learning/automated-agent-evaluation-with-amazon-bedrock-agentcore-and-github-actions/). The broader industry recommendation is to gate on a *regression* — block the PR only when scores drop more than 3% from baseline — rather than on an absolute number. Block on signal, not on noise. ## Trajectory Checks: Cheaper and More Deterministic Before reaching for LLM-as-judge for every check, consider trajectory evaluation. AgentCore includes three trajectory evaluators: **ExactOrderMatch**, **InOrderMatch**, and **AnyOrderMatch**. These validate that an agent called tool_X before tool_Y, or that a specific sequence of tools was used in the expected order. They are deterministic, add no extra model invocation costs, and catch a large class of multi-step agent regressions. Use trajectory checks as your primary gate; reserve LLM scoring for the subjective quality questions that trajectory checks cannot answer. [DeepEval’s LLM-as-judge analysis](https://deepeval.com/blog/llm-as-a-judge) covers the broader variance problem in depth if you want more context on when to use each approach. ## The Honest Trade-offs This pipeline is not free. A few things to plan around: - **10 minutes per PR.** CDK deployment, 30-second runtime startup, 30-90 seconds for CloudWatch trace propagation — it adds up. This is a PR check, not a pre-commit hook. - **Cost scales with evaluators.** Four evaluators across five test prompts means 20 LLM judge model calls per pull request. At moderate PR volume that is manageable; at high volume, trajectory checks reduce that number significantly. - **ARM64 cross-compilation.** The MCP server runtime requires ARM64 container images, but GitHub-hosted runners are x86_64. You will need Docker Buildx with QEMU cross-compilation. The reference implementation handles this, but be aware if you are customizing the Dockerfile. - **M2M tokens bypass role checks.** By design, machine-to-machine tokens used by the CI pipeline get access to all tools, skipping the role enforcement applied to real users. Flag this with your security team before deploying to a sensitive environment. ## Getting Started The minimal path to running agent evaluation in GitHub Actions, using the [agent-evaluation library](https://awslabs.github.io/agent-evaluation/cicd/): ``` name: Agent Evaluation on: pull_request: branches: [main] env: AWS_REGION: us-east-1 jobs: evaluate: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - uses: aws-actions/configure-aws-credentials@v4 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} aws-region: ${{ env.AWS_REGION }} - run: pip install agent-evaluation - run: agenteval run - name: Post Results run: cat agenteval_summary.md >> $GITHUB_STEP_SUMMARY ``` For the full stack — CDK infrastructure, Cognito setup, OIDC federation, and multi-evaluator scoring — follow AWS’s detailed implementation guide linked above. The complete reference implementation takes roughly a day to wire up the first time. ## AWS Is Not the Only Option If you are not in the AWS ecosystem, the same pattern exists elsewhere. [DeepEval](https://www.deepeval.com) offers a pytest-native interface with 50+ metrics that runs framework-agnostically. LangSmith integrates tightly with LangChain and LangGraph workloads. Langfuse provides a self-hosted open-source option for teams that need data residency control. The AWS implementation stands out for its managed infrastructure and native integration with IAM, CloudWatch, and CDK — if your agents already live in AWS, that coherence is worth something. The broader point: every major cloud now offers agent evaluation as a managed service. The tooling excuse for skipping agent testing is gone. Your agents deserve the same quality gates your APIs have had for a decade.