Amazon Bedrock AgentCore Evaluations can now use decision models as scorers. An AWS Machine Learning blog post from November 4, 2026, by Sungyeon Kim, Bharathi Srinivasan, Haibo Ding and Jonathan Buck, shows how to run them as code-based evaluators in AWS Lambda. Decision models return probabilities over a fixed set of answers, so they suit narrow, repeatable checks such as "did the response stay on policy?" or "does it match the retrieved context?" AWS recommends using them as a cheap screening layer and sending only low-confidence turns to an LLM judge.
The original is the AWS post on decision models with AgentCore Evaluations. The page we could read was truncated, so this guide covers what it states and flags what we could not see: pricing, service limits and full code listings.
TL;DR #
| Question | Answer |
|---|---|
| What is new? | Decision models as code-based evaluators in AgentCore Evaluations |
| How do they run? | In AWS Lambda; the Lambda turns OpenTelemetry spans into a request and maps probabilities back |
| Backends | Jev (hosted by TypeSafe, API key) or Strands Decider 2B (open source, self-hosted) |
| Question types | noul (yes/no),choice (up to 255 options),score (2 to 10 levels) |
| Suggested use | Screen every turn; escalate low-confidence turns to an LLM evaluator |
| Numbers published? | None in the text we read |
| Pricing and limits? | Not in the text we read |
What is AgentCore Evaluations? #
According to the AWS post, AgentCore Evaluations runs in three modes: on demand, in batch against stored sessions, and online against production traffic. It ships with 15 built-in evaluators and supports custom evaluators, either LLM-as-a-judge or code-based. The new piece is a pattern for the code-based kind: a Lambda function that calls a decision model.
If you are new to agent evals in general, our explainer on AI evals for engineers and PMs and the guide to coding agent evals on real repos give the background. AgentCore itself also appeared in our walkthrough of an aerospace part-substitution agent built with Nova embeddings, AgentCore and Strands.
What is a decision model? #
AWS describes decision models as models that take structured input and a fixed set of allowed answers, then return probabilities over those answers. They do not write explanations. They pick. That makes them fast and cheap compared with a full LLM judge, and it makes their output easy to threshold.
The post lists three question types:
| Type | Output | Example use |
|---|---|---|
noul |
Yes or no, probability between 0 and 1 | Is the answer grounded in the retrieved text? |
choice |
One of up to 255 options, with the full distribution and a confidence | Did the session complete the goal fully, partly or not at all? |
score |
One of 2 to 10 ordered levels, with a weighted score and confidence | How helpful was this turn on a four-level scale? |
For a broader tour of the field, see our ranking of the top decision models.
Which backends can you use? #
The sample supports two.
Jev. A hosted model from TypeSafe. You need a TypeSafe API key, which the sample reads from AWS Secrets Manager via a secret ARN. We have covered Jev from several angles: the speed and cost claims fact-check, cheap verification checkpoints in agent pipelines and Jev versus XGBoost and BERT classifiers.
Strands Decider 2B. An open-source model you host yourself, either on a local machine or on Amazon EC2 inside your VPC so Lambda can reach it. Our post Strands Decider 2B: open-source decision model covers the release.
The tradeoff is the usual one. A hosted model is quick to start but sends data to a third party and adds a key to manage. A self-hosted 2B model keeps data inside your VPC but gives you an EC2 instance, security groups and patching to run.
How the Lambda evaluator works #
In the AWS design, the Lambda function does three jobs:
- Receives the evaluation event from AgentCore, which includes OpenTelemetry spans for the agent session.
- Converts those spans into a decision-model request: the structured input plus the allowed answers.
- Calls the model, then maps the returned probabilities into an AgentCore evaluation result.
Because the evaluator is plain code, you can version it, test it locally and review it like any other function. Compare that with an LLM judge, whose behavior shifts with the prompt and the model version.
Setup steps from the post #
Prerequisites listed by AWS:
- An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West (Oregon).
- CloudWatch Transaction Search enabled.
- Amazon Bedrock model access.
- Python 3.12 or newer, the AWS CLI and Docker.
The sample repository has two scripts. deploy.py deploys an HR Assistant agent. evaluate.py deploys the evaluators and runs the evaluation modes. Flags control the backend:
python evaluate.py --with-jev --jev-secret-arn <ARN>
python evaluate.py --with-decider
python evaluate.py --with-decider --decider-ec2
The flags above come from the post's description. We did not run the scripts, so check the repository README for exact usage. The code lives in awslabs/agentcore-samples under 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/custom-code-based-evaluation.
The three sample evaluators #
Each backend provides the same three evaluators, prefixed Jev or Decider:
| Evaluator | Level | Question type | Value returned |
|---|---|---|---|
| Groundedness | Trace | noul |
P(grounded) |
| Helpfulness | Trace | score (4 levels) |
Expected level, scaled 0 to 1 |
| GoalCompletion | Session | choice (3 options) |
Probability-weighted value |
Trace-level evaluators score one turn. Session-level evaluators look at a whole conversation. That split matters when you pick thresholds: a groundedness miss is an alert; a goal-completion miss is a trend.
The screening pattern AWS recommends #
The post's main design advice is to layer the checks. A decision model runs on every turn because it is cheap. Only turns where the model's confidence is low go to a built-in LLM evaluator, which gives a detailed explanation.
This mirrors a pattern we have seen in other places: a small fast model gates a large slow one. It also pairs with human review. Our guide on when to let an agent run covers the human end of the same ladder. The tiers look like this:
- Decision model on every turn.
- LLM judge on low-confidence turns.
- Human review on the cases the judge still cannot settle.
Set the confidence threshold from data, not by feel. Start loose, review what lands in each tier for a week, and tighten.
How to validate before you trust it #
AWS recommends checking a decision-model evaluator against labeled traffic before relying on it:
- Build a labeled sample that includes edge cases.
- Get majority-vote labels from three human reviewers and track split-vote turns.
- Run the decision-model and built-in evaluators on the same sessions.
- Compare agreement rates, calibration (does confidence track accuracy?) and disagreements.
- Repeat whenever you change a question or a model version.
Calibration deserves attention. A model that says 0.9 and is right 60% of the time will make your screening layer pass bad turns. The post names calibration but gives no benchmark figures in the text we read, so measure your own.
Decision model or LLM judge: which one when? #
| Need | Decision model | LLM judge |
|---|---|---|
| Check every turn at low cost | Good fit | Expensive at volume |
| A written reason for a failure | None; it returns probabilities | Yes, in text |
| Fixed answer set (yes/no, a scale) | Designed for it | Works, but output can drift |
| Open-ended quality review | Weak fit | Better fit |
| Threshold alerts | Easy: use the probability | Needs parsing of the verdict |
A worked example with the HR Assistant sample: ask the groundedness question on every turn. If the probability of "grounded" is 0.95 or higher, log it and move on. If it falls between 0.5 and 0.95, hand the turn to a built-in LLM evaluator and keep its explanation for the on-call engineer. If it falls below 0.5, flag the turn for human review. These cut-offs are our illustration, not AWS values, so tune them on your labeled sample.
For reference material, AWS links the AgentCore Evaluations documentation, the Strands Decider announcement and TypeSafe's post introducing its System One models and Jev. Read those before you commit to a backend.
Security and data notes #
Agent traces can carry personal data, prompts and tool outputs. Before you send spans to a hosted backend, check what they contain. The self-hosted Strands Decider route keeps traffic inside your VPC. For runtime protection, evaluation is not enough: it scores behavior after the fact. AgentBeam, built by the explainx.ai team, is an agent security platform that stops AI agents before they take dangerous actions.
What is missing #
- Pricing for Jev calls, Lambda runs and EC2 hosting.
- Service limits for custom evaluators.
- Accuracy and latency numbers for either backend.
- Full code listings, which the page we read did not include.
Treat the pattern as a design to test, not a proven result. If you adopt it, record your own agreement and calibration numbers and share them with your team.
Related reading #
Details come from the AWS Machine Learning blog post of November 4, 2026 and may change as AWS updates the sample and docs.