Developers are shipping AI agents to production without tests. Not because they are reckless — because the tooling was not there. That changes now. AWS published a reference implementation that wires Amazon Bedrock AgentCore Evaluations into GitHub Actions, turning every pull request that touches an agent into a gated quality check. If your agent’s goal success rate drops, the merge is blocked. Same as your unit tests. It took the software industry a decade to build CI/CD culture around code; AI agents are about to get the same treatment.
What AWS Shipped #
Amazon Bedrock AgentCore Evaluations went generally available in March 2026. AWS’s new reference implementation extends it into pull request workflows: when a PR modifies agent code, a system prompt, a model selection, or a tool configuration, GitHub Actions automatically deploys the agent to a dev environment, invokes it with a test dataset, retrieves the resulting OpenTelemetry traces from CloudWatch, scores them using AgentCore’s built-in evaluators, and blocks the merge if any score falls below a configured threshold. The full reference implementation is available on GitHub.
The Four Evaluators That Gate Your PRs #
The reference implementation uses four evaluators from AgentCore’s catalog of 20+:
- GoalSuccessRate — Did the agent actually complete the task the user gave it?
- Correctness — Is the response accurate and appropriate given the prompt?
- ToolSelectionAccuracy — Did the agent pick the right tool from its available set?
- ToolParameterAccuracy — Did it derive the right parameters from context before calling the tool?
This matters because most AI monitoring only checks the final output. These evaluators examine the reasoning path — whether the agent reached a good answer through a correct process or stumbled onto it by accident. The broader catalog also includes safety evaluators (Harmfulness, Stereotyping, Refusal) and trajectory checks for validating multi-step tool sequences.
The Variance Problem Nobody Talks About #
Here is the implementation detail that will save you a frustrating afternoon: LLM-as-judge evaluation is non-deterministic. Run the same agent against the same prompts twice, and you might see scores of 0.84 and 0.79 with zero code changes. Set a hard threshold at 0.80 and you will generate random red builds that erode team confidence in the pipeline.
The fix is straightforward: set thresholds with margin below your target reliability, and consider averaging across multiple evaluation runs before applying a pass/fail decision. AWS explicitly warns about this in their implementation guide. The broader industry recommendation is to gate on a regression — block the PR only when scores drop more than 3% from baseline — rather than on an absolute number. Block on signal, not on noise.
Trajectory Checks: Cheaper and More Deterministic #
Before reaching for LLM-as-judge for every check, consider trajectory evaluation. AgentCore includes three trajectory evaluators: ExactOrderMatch, InOrderMatch, and AnyOrderMatch. These validate that an agent called tool_X before tool_Y, or that a specific sequence of tools was used in the expected order. They are deterministic, add no extra model invocation costs, and catch a large class of multi-step agent regressions. Use trajectory checks as your primary gate; reserve LLM scoring for the subjective quality questions that trajectory checks cannot answer. DeepEval’s LLM-as-judge analysis covers the broader variance problem in depth if you want more context on when to use each approach.
The Honest Trade-offs #
This pipeline is not free. A few things to plan around:
- 10 minutes per PR. CDK deployment, 30-second runtime startup, 30-90 seconds for CloudWatch trace propagation — it adds up. This is a PR check, not a pre-commit hook.
- Cost scales with evaluators. Four evaluators across five test prompts means 20 LLM judge model calls per pull request. At moderate PR volume that is manageable; at high volume, trajectory checks reduce that number significantly.
- ARM64 cross-compilation. The MCP server runtime requires ARM64 container images, but GitHub-hosted runners are x86_64. You will need Docker Buildx with QEMU cross-compilation. The reference implementation handles this, but be aware if you are customizing the Dockerfile.
- M2M tokens bypass role checks. By design, machine-to-machine tokens used by the CI pipeline get access to all tools, skipping the role enforcement applied to real users. Flag this with your security team before deploying to a sensitive environment.
Getting Started #
The minimal path to running agent evaluation in GitHub Actions, using the agent-evaluation library:
name: Agent Evaluation
on:
pull_request:
branches: [main]
env:
AWS_REGION: us-east-1
jobs:
evaluate:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: ${{ env.AWS_REGION }}
- run: pip install agent-evaluation
- run: agenteval run
- name: Post Results
run: cat agenteval_summary.md >> $GITHUB_STEP_SUMMARY
For the full stack — CDK infrastructure, Cognito setup, OIDC federation, and multi-evaluator scoring — follow AWS’s detailed implementation guide linked above. The complete reference implementation takes roughly a day to wire up the first time.
AWS Is Not the Only Option #
If you are not in the AWS ecosystem, the same pattern exists elsewhere. DeepEval offers a pytest-native interface with 50+ metrics that runs framework-agnostically. LangSmith integrates tightly with LangChain and LangGraph workloads. Langfuse provides a self-hosted open-source option for teams that need data residency control. The AWS implementation stands out for its managed infrastructure and native integration with IAM, CloudWatch, and CDK — if your agents already live in AWS, that coherence is worth something.
The broader point: every major cloud now offers agent evaluation as a managed service. The tooling excuse for skipping agent testing is gone. Your agents deserve the same quality gates your APIs have had for a decade.