Stop Guessing Which Model Is Better: Amazon Bedrock Model Evaluation Hands-On

Amazon Bedrock Model Evaluation enables teams to compare foundation models with data instead of guesswork, offering automatic metrics and LLM-as-a-Judge modes. A hands-on guide demonstrates building a dataset, running evaluation jobs, and analyzing results with command-line tools, emphasizing reproducibility and regression testing for model updates.

"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships. In this hands-on, we'll answer the question with data: Amazon Bedrock Model Evaluation , run in two modes — automatic metrics scored against reference answers, and LLM-as-a-Judge , where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with jq and awk . Prefer video? This entire hands-on is also on YouTube: Automatic evaluation runs your dataset through the target model and scores each response against your referenceResponse with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers. LLM-as-a-Judge has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model. Run both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security. Note on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names. All of this runs in CloudShell: export REGION=us-east-1 export ACCOUNT ID=$ aws sts get-caller-identity --query Account --output text export EVAL BUCKET="bedrock-eval-$ACCOUNT ID" aws s3 mb "s3://$EVAL BUCKET" --region $REGION The dataset is JSONL — one JSON object per line, each with a prompt and a referenceResponse the answer you consider correct . Ten AWS-basics Q&A pairs: cat /tmp/eval-dataset.jsonl << 'EOF' {"prompt":"Explain the main use of AWS Lambda in one sentence","referenceResponse":"A compute service that runs code serverlessly in response to events"} {"prompt":"What kind of service is Amazon S3?","referenceResponse":"A highly available and highly durable object storage service"} {"prompt":"What kind of database is Amazon DynamoDB?","referenceResponse":"A fully managed NoSQL key-value database"} {"prompt":"What is a CloudWatch alarm?","referenceResponse":"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold"} {"prompt":"What is an IAM role?","referenceResponse":"A mechanism for granting temporary permissions to AWS resources and users"} {"prompt":"What is Amazon VPC?","referenceResponse":"A service for building a logically isolated virtual network on AWS"} {"prompt":"What are the characteristics of Amazon RDS?","referenceResponse":"A service that runs relational databases in a fully managed way"} {"prompt":"What is the main function of CloudFront?","referenceResponse":"A content delivery network CDN that uses edge locations"} {"prompt":"What kind of service is Amazon SQS?","referenceResponse":"A managed message queuing service"} {"prompt":"What is Amazon Bedrock?","referenceResponse":"A service that provides foundation models through a serverless, unified API"} EOF aws s3 cp /tmp/eval-dataset.jsonl "s3://$EVAL BUCKET/input/dataset.jsonl" Ten pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file is the asset worth investing in: it becomes your regression test for every future model release. Bedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models: cat /tmp/eval-trust.json << 'EOF' {"Version":"2012-10-17","Statement": {"Effect":"Allow","Principal":{"Service":"bedrock.amazonaws.com"},"Action":"sts:AssumeRole"} } EOF EVAL ROLE ARN=$ aws iam create-role \ --role-name BedrockEvaluationRole \ --assume-role-policy-document file:///tmp/eval-trust.json \ --query 'Role.Arn' --output text cat /tmp/eval-policy.json << EOF { "Version": "2012-10-17", "Statement": {"Effect":"Allow","Action": "s3:GetObject","s3:ListBucket","s3:PutObject" , "Resource": "arn:aws:s3:::$EVAL BUCKET","arn:aws:s3:::$EVAL BUCKET/ " }, {"Effect":"Allow","Action": "bedrock:InvokeModel" , "Resource": "arn:aws:bedrock: ::foundation-model/ ","arn:aws:bedrock: : :inference-profile/ " } } EOF aws iam put-role-policy --role-name BedrockEvaluationRole \ --policy-name inline --policy-document file:///tmp/eval-policy.json Note the trust policy: the principal is bedrock.amazonaws.com — this role is for the service, not for you or a Lambda. In the Bedrock console Inference and Assessment → Evaluations , create two jobs against the same dataset — the video above walks through every screen: s3://$EVAL BUCKET/input/dataset.jsonl , the BedrockEvaluationRole , and an output path under s3://$EVAL BUCKET/output/auto/ s3://$EVAL BUCKET/output/judge/ Both jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size. When the jobs complete, sync everything down and find the result files: aws s3 sync "s3://$EVAL BUCKET/output/" /tmp/eval-out/ find /tmp/eval-out/ -type f List the main output files find /tmp/eval-out/ -name " output.jsonl" -not -name " .out" The output is deeply nested JSONL — one record per prompt, each carrying a scores array. Aggregate the automatic metrics into per-metric count / mean / max: for f in /tmp/eval-out/auto/ / /models/ /taskTypes/ /datasets/ / output.jsonl; do cat "$f" | jq -r '.automatedEvaluationResult.scores | "\ .metricName \t\ .result "' done | awk -F'\t' ' {sum $1 +=$2; cnt $1 ++; if $2 max $1 max $1 =$2} END {for m in sum printf "%s\tcount %d\tmean %.4f\tmax %.4f\n", m, cnt m , sum m /cnt m , max m } ' And the judge's scores, averaged per metric: JUDGE=$ find /tmp/eval-out/judge -name " output.jsonl" -not -name " .out" | head -1 cat "$JUDGE" | jq -r '.automatedEvaluationResult.scores | "\ .metricName \t\ .result "' | \ awk '{sum $1 +=$2; cnt $1 ++} END {for m in sum printf "%s\tmean %.2f\n", m, sum m /cnt m }' Two numbers-reading tips: S3 aws s3 rm "s3://$EVAL BUCKET" --recursive aws s3 rb "s3://$EVAL BUCKET" IAM aws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline aws iam delete-role --role-name BedrockEvaluationRole Model selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade. About the author Maruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain AWS / Google Cloud / Azure | Udemy instructor 100K+ students 🎥 Video version of this hands-on: https://youtu.be/jbrFKA34hWc https://youtu.be/jbrFKA34hWc 📚 Full course — AWS Certified Generative AI Developer Professional AIP-C01 Exam Prep: https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/ https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/ 👨🏫 All my courses: Eng https://www.udemy.com/user/maruchin-tech-2/ https://www.udemy.com/user/maruchin-tech-2/ Jpn https://www.udemy.com/user/shan-wang-wan-jun-2/ https://www.udemy.com/user/shan-wang-wan-jun-2/ 🎫 Monthly discount coupons: https://www.youtube.com/@MaruchinTech-cloud/posts https://www.youtube.com/@MaruchinTech-cloud/posts