"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships.
In this hands-on, we'll answer the question with data: Amazon Bedrock Model Evaluation, run in two modes — automatic metrics scored against reference answers, and LLM-as-a-Judge, where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with jq
and awk
.
Prefer video? This entire hands-on is also on YouTube:
Automatic evaluation runs your dataset through the target model and scores each response against your referenceResponse
with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers.
LLM-as-a-Judge has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model.
Run both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security.
Note on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names.
All of this runs in CloudShell:
export REGION=us-east-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export EVAL_BUCKET="bedrock-eval-$ACCOUNT_ID"
aws s3 mb "s3://$EVAL_BUCKET" --region $REGION
The dataset is JSONL — one JSON object per line, each with a prompt
and a referenceResponse
(the answer you consider correct). Ten AWS-basics Q&A pairs:
cat > /tmp/eval-dataset.jsonl << 'EOF'
{"prompt":"Explain the main use of AWS Lambda in one sentence","referenceResponse":"A compute service that runs code serverlessly in response to events"}
{"prompt":"What kind of service is Amazon S3?","referenceResponse":"A highly available and highly durable object storage service"}
{"prompt":"What kind of database is Amazon DynamoDB?","referenceResponse":"A fully managed NoSQL key-value database"}
{"prompt":"What is a CloudWatch alarm?","referenceResponse":"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold"}
{"prompt":"What is an IAM role?","referenceResponse":"A mechanism for granting temporary permissions to AWS resources and users"}
{"prompt":"What is Amazon VPC?","referenceResponse":"A service for building a logically isolated virtual network on AWS"}
{"prompt":"What are the characteristics of Amazon RDS?","referenceResponse":"A service that runs relational databases in a fully managed way"}
{"prompt":"What is the main function of CloudFront?","referenceResponse":"A content delivery network (CDN) that uses edge locations"}
{"prompt":"What kind of service is Amazon SQS?","referenceResponse":"A managed message queuing service"}
{"prompt":"What is Amazon Bedrock?","referenceResponse":"A service that provides foundation models through a serverless, unified API"}
EOF
aws s3 cp /tmp/eval-dataset.jsonl "s3://$EVAL_BUCKET/input/dataset.jsonl"
Ten pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file is the asset worth investing in: it becomes your regression test for every future model release.
Bedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models:
cat > /tmp/eval-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"bedrock.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF
EVAL_ROLE_ARN=$(aws iam create-role \
--role-name BedrockEvaluationRole \
--assume-role-policy-document file:///tmp/eval-trust.json \
--query 'Role.Arn' --output text)
cat > /tmp/eval-policy.json << EOF
{
"Version": "2012-10-17",
"Statement": [
{"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket","s3:PutObject"],
"Resource":["arn:aws:s3:::$EVAL_BUCKET","arn:aws:s3:::$EVAL_BUCKET/*"]},
{"Effect":"Allow","Action":["bedrock:InvokeModel"],
"Resource":["arn:aws:bedrock:*::foundation-model/*","arn:aws:bedrock:*:*:inference-profile/*"]}
]
}
EOF
aws iam put-role-policy --role-name BedrockEvaluationRole \
--policy-name inline --policy-document file:///tmp/eval-policy.json
Note the trust policy: the principal is bedrock.amazonaws.com
— this role is for the service, not for you or a Lambda.
In the Bedrock console (Inference and Assessment → Evaluations), create two jobs against the same dataset — the video above walks through every screen:
s3://$EVAL_BUCKET/input/dataset.jsonl
, the BedrockEvaluationRole
, and an output path under s3://$EVAL_BUCKET/output/auto/
s3://$EVAL_BUCKET/output/judge/
Both jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size.
When the jobs complete, sync everything down and find the result files:
aws s3 sync "s3://$EVAL_BUCKET/output/" /tmp/eval-out/
find /tmp/eval-out/ -type f
find /tmp/eval-out/ -name "*_output.jsonl" -not -name "*.out"
The output is deeply nested JSONL — one record per prompt, each carrying a scores
array. Aggregate the automatic metrics into per-metric count / mean / max:
for f in /tmp/eval-out/auto/*/*/models/*/taskTypes/*/datasets/*/*_output.jsonl; do
cat "$f" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"'
done | awk -F'\t' '
{sum[$1]+=$2; cnt[$1]++; if($2>max[$1]) max[$1]=$2}
END {for (m in sum) printf "%s\tcount %d\tmean %.4f\tmax %.4f\n", m, cnt[m], sum[m]/cnt[m], max[m]}
'
And the judge's scores, averaged per metric:
JUDGE=$(find /tmp/eval-out/judge -name "*_output.jsonl" -not -name "*.out" | head -1)
cat "$JUDGE" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"' | \
awk '{sum[$1]+=$2; cnt[$1]++} END {for (m in sum) printf "%s\tmean %.2f\n", m, sum[m]/cnt[m]}'
Two numbers-reading tips:
aws s3 rm "s3://$EVAL_BUCKET" --recursive
aws s3 rb "s3://$EVAL_BUCKET"
aws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline
aws iam delete-role --role-name BedrockEvaluationRole
Model selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade.
About the author
Maruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain (AWS / Google Cloud / Azure) | Udemy instructor (100K+ students)
🎥 Video version of this hands-on:
📚 Full course — AWS Certified Generative AI Developer Professional (AIP-C01) Exam Prep:
https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/
👨🏫 All my courses:
(Eng) https://www.udemy.com/user/maruchin-tech-2/
(Jpn) https://www.udemy.com/user/shan-wang-wan-jun-2/
🎫 Monthly discount coupons:
https://www.youtube.com/@MaruchinTech-cloud/posts