# Stop Guessing Which Model Is Better: Amazon Bedrock Model Evaluation Hands-On

> Source: <https://dev.to/maruchin_tech_555/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on-253k>
> Published: 2026-08-22 08:17:00+00:00

"Which model should we use?" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships.

In this hands-on, we'll answer the question with data: **Amazon Bedrock Model Evaluation**, run in two modes — automatic metrics scored against reference answers, and **LLM-as-a-Judge**, where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with `jq`

and `awk`

.

Prefer video? This entire hands-on is also on YouTube:

**Automatic evaluation** runs your dataset through the target model and scores each response against your `referenceResponse`

with built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers.

**LLM-as-a-Judge** has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model.

Run both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security.

Note on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names.

All of this runs in CloudShell:

```
export REGION=us-east-1
export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
export EVAL_BUCKET="bedrock-eval-$ACCOUNT_ID"

aws s3 mb "s3://$EVAL_BUCKET" --region $REGION
```

The dataset is JSONL — one JSON object per line, each with a `prompt`

and a `referenceResponse`

(the answer you consider correct). Ten AWS-basics Q&A pairs:

```
cat > /tmp/eval-dataset.jsonl << 'EOF'
{"prompt":"Explain the main use of AWS Lambda in one sentence","referenceResponse":"A compute service that runs code serverlessly in response to events"}
{"prompt":"What kind of service is Amazon S3?","referenceResponse":"A highly available and highly durable object storage service"}
{"prompt":"What kind of database is Amazon DynamoDB?","referenceResponse":"A fully managed NoSQL key-value database"}
{"prompt":"What is a CloudWatch alarm?","referenceResponse":"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold"}
{"prompt":"What is an IAM role?","referenceResponse":"A mechanism for granting temporary permissions to AWS resources and users"}
{"prompt":"What is Amazon VPC?","referenceResponse":"A service for building a logically isolated virtual network on AWS"}
{"prompt":"What are the characteristics of Amazon RDS?","referenceResponse":"A service that runs relational databases in a fully managed way"}
{"prompt":"What is the main function of CloudFront?","referenceResponse":"A content delivery network (CDN) that uses edge locations"}
{"prompt":"What kind of service is Amazon SQS?","referenceResponse":"A managed message queuing service"}
{"prompt":"What is Amazon Bedrock?","referenceResponse":"A service that provides foundation models through a serverless, unified API"}
EOF

aws s3 cp /tmp/eval-dataset.jsonl "s3://$EVAL_BUCKET/input/dataset.jsonl"
```

Ten pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file *is* the asset worth investing in: it becomes your regression test for every future model release.

Bedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models:

```
cat > /tmp/eval-trust.json << 'EOF'
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"Service":"bedrock.amazonaws.com"},"Action":"sts:AssumeRole"}]}
EOF

EVAL_ROLE_ARN=$(aws iam create-role \
  --role-name BedrockEvaluationRole \
  --assume-role-policy-document file:///tmp/eval-trust.json \
  --query 'Role.Arn' --output text)

cat > /tmp/eval-policy.json << EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect":"Allow","Action":["s3:GetObject","s3:ListBucket","s3:PutObject"],
     "Resource":["arn:aws:s3:::$EVAL_BUCKET","arn:aws:s3:::$EVAL_BUCKET/*"]},
    {"Effect":"Allow","Action":["bedrock:InvokeModel"],
     "Resource":["arn:aws:bedrock:*::foundation-model/*","arn:aws:bedrock:*:*:inference-profile/*"]}
  ]
}
EOF

aws iam put-role-policy --role-name BedrockEvaluationRole \
  --policy-name inline --policy-document file:///tmp/eval-policy.json
```

Note the trust policy: the principal is `bedrock.amazonaws.com`

— this role is for the service, not for you or a Lambda.

In the Bedrock console (Inference and Assessment → Evaluations), create two jobs against the same dataset — the video above walks through every screen:

`s3://$EVAL_BUCKET/input/dataset.jsonl`

, the `BedrockEvaluationRole`

, and an output path under `s3://$EVAL_BUCKET/output/auto/`

`s3://$EVAL_BUCKET/output/judge/`

Both jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size.

When the jobs complete, sync everything down and find the result files:

```
aws s3 sync "s3://$EVAL_BUCKET/output/" /tmp/eval-out/
find /tmp/eval-out/ -type f

# List the main output files
find /tmp/eval-out/ -name "*_output.jsonl" -not -name "*.out"
```

The output is deeply nested JSONL — one record per prompt, each carrying a `scores`

array. Aggregate the automatic metrics into per-metric count / mean / max:

```
for f in /tmp/eval-out/auto/*/*/models/*/taskTypes/*/datasets/*/*_output.jsonl; do
  cat "$f" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"'
done | awk -F'\t' '
  {sum[$1]+=$2; cnt[$1]++; if($2>max[$1]) max[$1]=$2}
  END {for (m in sum) printf "%s\tcount %d\tmean %.4f\tmax %.4f\n", m, cnt[m], sum[m]/cnt[m], max[m]}
'
```

And the judge's scores, averaged per metric:

```
JUDGE=$(find /tmp/eval-out/judge -name "*_output.jsonl" -not -name "*.out" | head -1)

cat "$JUDGE" | jq -r '.automatedEvaluationResult.scores[] | "\(.metricName)\t\(.result)"' | \
  awk '{sum[$1]+=$2; cnt[$1]++} END {for (m in sum) printf "%s\tmean %.2f\n", m, sum[m]/cnt[m]}'
```

Two numbers-reading tips:

```
# S3
aws s3 rm "s3://$EVAL_BUCKET" --recursive
aws s3 rb "s3://$EVAL_BUCKET"

# IAM
aws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline
aws iam delete-role --role-name BedrockEvaluationRole
```

Model selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade.

**About the author**

Maruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain (AWS / Google Cloud / Azure) | Udemy instructor (100K+ students)

🎥 Video version of this hands-on:

[https://youtu.be/jbrFKA34hWc](https://youtu.be/jbrFKA34hWc)

📚 Full course — AWS Certified Generative AI Developer Professional (AIP-C01) Exam Prep:

[https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/](https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/)

👨🏫 All my courses:

(Eng) [https://www.udemy.com/user/maruchin-tech-2/](https://www.udemy.com/user/maruchin-tech-2/)

(Jpn) [https://www.udemy.com/user/shan-wang-wan-jun-2/](https://www.udemy.com/user/shan-wang-wan-jun-2/)

🎫 Monthly discount coupons:

[https://www.youtube.com/@MaruchinTech-cloud/posts](https://www.youtube.com/@MaruchinTech-cloud/posts)