AgentCore Evaluations Now Accepts Decision Models: Cheap Per-Turn Checks with Jev or Strands Decider

Amazon Bedrock AgentCore Evaluations now accepts decision models as code-based evaluators, according to a November 4, 2026 AWS Machine Learning blog post by Sungyeon Kim, Bharathi Srinivasan, Haibo Ding and Jonathan Buck. The decision models run in AWS Lambda, return probabilities over a fixed set of answers across three question types — `noul` (yes/no), `choice` (up to 255 options) and `score` (2 to 10 ordered levels) — and are backed by either TypeSafe's hosted Jev model or the open-source, self-hosted Strands Decider 2B. AWS recommends using the decision models as a cheap screening layer on every turn and escalating only low-confidence turns to an LLM judge; the post published no pricing, service limits or performance numbers.

Amazon Bedrock AgentCore Evaluations can now use decision models as scorers. An AWS Machine Learning blog post from November 4, 2026, by Sungyeon Kim, Bharathi Srinivasan, Haibo Ding and Jonathan Buck, shows how to run them as code-based evaluators in AWS Lambda. Decision models return probabilities over a fixed set of answers, so they suit narrow, repeatable checks such as "did the response stay on policy?" or "does it match the retrieved context?" AWS recommends using them as a cheap screening layer and sending only low-confidence turns to an LLM judge. The original is the AWS post on decision models with AgentCore Evaluations https://aws.amazon.com/blogs/machine-learning/use-decision-models-with-amazon-bedrock-agentcore-evaluations/?ref=explainx . The page we could read was truncated, so this guide covers what it states and flags what we could not see: pricing, service limits and full code listings. TL;DR | Question | Answer | |---|---| | What is new? | Decision models as code-based evaluators in AgentCore Evaluations | | How do they run? | In AWS Lambda; the Lambda turns OpenTelemetry spans into a request and maps probabilities back | | Backends | Jev hosted by TypeSafe, API key or Strands Decider 2B open source, self-hosted | | Question types | noul yes/no , choice up to 255 options , score 2 to 10 levels | | Suggested use | Screen every turn; escalate low-confidence turns to an LLM evaluator | | Numbers published? | None in the text we read | | Pricing and limits? | Not in the text we read | What is AgentCore Evaluations? According to the AWS post, AgentCore Evaluations runs in three modes: on demand, in batch against stored sessions, and online against production traffic. It ships with 15 built-in evaluators and supports custom evaluators, either LLM-as-a-judge or code-based. The new piece is a pattern for the code-based kind: a Lambda function that calls a decision model. If you are new to agent evals in general, our explainer on AI evals for engineers and PMs https://explainx.ai/blog/ai-evals-for-engineers-and-pms-explained-2026 and the guide to coding agent evals on real repos https://explainx.ai/blog/ai-coding-agent-evals-real-repos-2026 give the background. AgentCore itself also appeared in our walkthrough of an aerospace part-substitution agent built with Nova embeddings, AgentCore and Strands https://explainx.ai/blog/aws-nova-multimodal-embeddings-agentcore-strands-aerospace-part-substitution-agent-36-query-eval-guide-2026 . What is a decision model? AWS describes decision models as models that take structured input and a fixed set of allowed answers, then return probabilities over those answers. They do not write explanations. They pick. That makes them fast and cheap compared with a full LLM judge, and it makes their output easy to threshold. The post lists three question types: | Type | Output | Example use | |---|---|---| | noul | Yes or no, probability between 0 and 1 | Is the answer grounded in the retrieved text? | | choice | One of up to 255 options, with the full distribution and a confidence | Did the session complete the goal fully, partly or not at all? | | score | One of 2 to 10 ordered levels, with a weighted score and confidence | How helpful was this turn on a four-level scale? | For a broader tour of the field, see our ranking of the top decision models https://explainx.ai/blog/top-10-decision-models-jev-perplexity-openai-open-source-ranked-2026 . Which backends can you use? The sample supports two. Jev. A hosted model from TypeSafe. You need a TypeSafe API key, which the sample reads from AWS Secrets Manager via a secret ARN. We have covered Jev from several angles: the speed and cost claims fact-check https://explainx.ai/blog/jev-speed-cost-claims-fact-check-2026 , cheap verification checkpoints in agent pipelines https://explainx.ai/blog/jev-cheap-verification-checkpoints-agent-pipelines-2026 and Jev versus XGBoost and BERT classifiers https://explainx.ai/blog/jev-vs-xgboost-bert-classifiers-2026 . Strands Decider 2B. An open-source model you host yourself, either on a local machine or on Amazon EC2 inside your VPC so Lambda can reach it. Our post Strands Decider 2B: open-source decision model https://explainx.ai/blog/strands-decider-2b-open-source-decision-model-2026 covers the release. The tradeoff is the usual one. A hosted model is quick to start but sends data to a third party and adds a key to manage. A self-hosted 2B model keeps data inside your VPC but gives you an EC2 instance, security groups and patching to run. How the Lambda evaluator works In the AWS design, the Lambda function does three jobs: 1. Receives the evaluation event from AgentCore, which includes OpenTelemetry spans for the agent session. 2. Converts those spans into a decision-model request: the structured input plus the allowed answers. 3. Calls the model, then maps the returned probabilities into an AgentCore evaluation result. Because the evaluator is plain code, you can version it, test it locally and review it like any other function. Compare that with an LLM judge, whose behavior shifts with the prompt and the model version. Setup steps from the post Prerequisites listed by AWS: - An AWS account in a Region where AgentCore Evaluations is available. The sample defaults to US West Oregon . - CloudWatch Transaction Search enabled. - Amazon Bedrock model access. - Python 3.12 or newer, the AWS CLI and Docker. The sample repository has two scripts. deploy.py deploys an HR Assistant agent. evaluate.py deploys the evaluators and runs the evaluation modes. Flags control the backend: Jev backend python evaluate.py --with-jev --jev-secret-arn <ARN Strands Decider backend python evaluate.py --with-decider Strands Decider on EC2 with VPC wiring python evaluate.py --with-decider --decider-ec2 The flags above come from the post's description. We did not run the scripts, so check the repository README for exact usage. The code lives in awslabs/agentcore-samples under 01-features/06-observe-evaluate-optimize-your-agent/02-evaluate/custom-code-based-evaluation . The three sample evaluators Each backend provides the same three evaluators, prefixed Jev or Decider : | Evaluator | Level | Question type | Value returned | |---|---|---|---| | Groundedness | Trace | noul | P grounded | | Helpfulness | Trace | score 4 levels | Expected level, scaled 0 to 1 | | GoalCompletion | Session | choice 3 options | Probability-weighted value | Trace-level evaluators score one turn. Session-level evaluators look at a whole conversation. That split matters when you pick thresholds: a groundedness miss is an alert; a goal-completion miss is a trend. The screening pattern AWS recommends The post's main design advice is to layer the checks. A decision model runs on every turn because it is cheap. Only turns where the model's confidence is low go to a built-in LLM evaluator, which gives a detailed explanation. This mirrors a pattern we have seen in other places: a small fast model gates a large slow one. It also pairs with human review. Our guide on when to let an agent run https://explainx.ai/blog/human-in-the-loop-ai-when-to-let-agent-run-2026 covers the human end of the same ladder. The tiers look like this: 1. Decision model on every turn. 2. LLM judge on low-confidence turns. 3. Human review on the cases the judge still cannot settle. Set the confidence threshold from data, not by feel. Start loose, review what lands in each tier for a week, and tighten. How to validate before you trust it AWS recommends checking a decision-model evaluator against labeled traffic before relying on it: 1. Build a labeled sample that includes edge cases. 2. Get majority-vote labels from three human reviewers and track split-vote turns. 3. Run the decision-model and built-in evaluators on the same sessions. 4. Compare agreement rates, calibration does confidence track accuracy? and disagreements. 5. Repeat whenever you change a question or a model version. Calibration deserves attention. A model that says 0.9 and is right 60% of the time will make your screening layer pass bad turns. The post names calibration but gives no benchmark figures in the text we read, so measure your own. Decision model or LLM judge: which one when? | Need | Decision model | LLM judge | |---|---|---| | Check every turn at low cost | Good fit | Expensive at volume | | A written reason for a failure | None; it returns probabilities | Yes, in text | | Fixed answer set yes/no, a scale | Designed for it | Works, but output can drift | | Open-ended quality review | Weak fit | Better fit | | Threshold alerts | Easy: use the probability | Needs parsing of the verdict | A worked example with the HR Assistant sample: ask the groundedness question on every turn. If the probability of "grounded" is 0.95 or higher, log it and move on. If it falls between 0.5 and 0.95, hand the turn to a built-in LLM evaluator and keep its explanation for the on-call engineer. If it falls below 0.5, flag the turn for human review. These cut-offs are our illustration, not AWS values, so tune them on your labeled sample. For reference material, AWS links the AgentCore Evaluations documentation https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/evaluations.html?ref=explainx , the Strands Decider announcement https://strandsagents.com/blog/introducing-strands-decider/?ref=explainx and TypeSafe's post introducing its System One models and Jev https://typesafe.ai/blog/introducing-system-one-models-and-jev?ref=explainx . Read those before you commit to a backend. Security and data notes Agent traces can carry personal data, prompts and tool outputs. Before you send spans to a hosted backend, check what they contain. The self-hosted Strands Decider route keeps traffic inside your VPC. For runtime protection, evaluation is not enough: it scores behavior after the fact. AgentBeam https://agentbeam.com/?ref=explainx , built by the explainx.ai team, is an agent security platform that stops AI agents before they take dangerous actions. What is missing - Pricing for Jev calls, Lambda runs and EC2 hosting. - Service limits for custom evaluators. - Accuracy and latency numbers for either backend. - Full code listings, which the page we read did not include. Treat the pattern as a design to test, not a proven result. If you adopt it, record your own agreement and calibration numbers and share them with your team. Related reading Details come from the AWS Machine Learning blog post of November 4, 2026 and may change as AWS updates the sample and docs.