{"slug": "stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on", "title": "Stop Guessing Which Model Is Better: Amazon Bedrock Model Evaluation Hands-On", "summary": "Amazon Bedrock Model Evaluation enables teams to compare foundation models with data instead of guesswork, offering automatic metrics and LLM-as-a-Judge modes. A hands-on guide demonstrates building a dataset, running evaluation jobs, and analyzing results with command-line tools, emphasizing reproducibility and regression testing for model updates.", "body_md": "\"Which model should we use?\" is the most common question in every Bedrock project — and most teams answer it by eyeballing a few responses. That doesn't scale, it isn't reproducible, and it silently expires every time a new model version ships.\n\nIn this hands-on, we'll answer the question with data: **Amazon Bedrock Model Evaluation**, run in two modes — automatic metrics scored against reference answers, and **LLM-as-a-Judge**, where a stronger model grades each response. We'll build the evaluation dataset, run the jobs, and crunch the result files down to comparable numbers with `jq`\n\nand `awk`\n\n.\n\nPrefer video? This entire hands-on is also on YouTube:\n\n**Automatic evaluation** runs your dataset through the target model and scores each response against your `referenceResponse`\n\nwith built-in metrics — accuracy-style similarity scores, robustness, toxicity. Fast, cheap, objective, but only as good as your reference answers.\n\n**LLM-as-a-Judge** has a judge model read each prompt/response pair and grade qualities like correctness, completeness, and helpfulness. It catches what string-similarity metrics can't — a response can be worded completely differently from the reference and still be right — at the cost of running a second, stronger model.\n\nRun both and you get two independent views of the same model, the same defense-in-depth idea applied to quality instead of security.\n\nNote on model IDs: Bedrock models are updated frequently — pick current models when you create the evaluation jobs, not whatever a months-old article names.\n\nAll of this runs in CloudShell:\n\n```\nexport REGION=us-east-1\nexport ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)\nexport EVAL_BUCKET=\"bedrock-eval-$ACCOUNT_ID\"\n\naws s3 mb \"s3://$EVAL_BUCKET\" --region $REGION\n```\n\nThe dataset is JSONL — one JSON object per line, each with a `prompt`\n\nand a `referenceResponse`\n\n(the answer you consider correct). Ten AWS-basics Q&A pairs:\n\n```\ncat > /tmp/eval-dataset.jsonl << 'EOF'\n{\"prompt\":\"Explain the main use of AWS Lambda in one sentence\",\"referenceResponse\":\"A compute service that runs code serverlessly in response to events\"}\n{\"prompt\":\"What kind of service is Amazon S3?\",\"referenceResponse\":\"A highly available and highly durable object storage service\"}\n{\"prompt\":\"What kind of database is Amazon DynamoDB?\",\"referenceResponse\":\"A fully managed NoSQL key-value database\"}\n{\"prompt\":\"What is a CloudWatch alarm?\",\"referenceResponse\":\"A mechanism that sends notifications or triggers automated actions when a metric crosses a threshold\"}\n{\"prompt\":\"What is an IAM role?\",\"referenceResponse\":\"A mechanism for granting temporary permissions to AWS resources and users\"}\n{\"prompt\":\"What is Amazon VPC?\",\"referenceResponse\":\"A service for building a logically isolated virtual network on AWS\"}\n{\"prompt\":\"What are the characteristics of Amazon RDS?\",\"referenceResponse\":\"A service that runs relational databases in a fully managed way\"}\n{\"prompt\":\"What is the main function of CloudFront?\",\"referenceResponse\":\"A content delivery network (CDN) that uses edge locations\"}\n{\"prompt\":\"What kind of service is Amazon SQS?\",\"referenceResponse\":\"A managed message queuing service\"}\n{\"prompt\":\"What is Amazon Bedrock?\",\"referenceResponse\":\"A service that provides foundation models through a serverless, unified API\"}\nEOF\n\naws s3 cp /tmp/eval-dataset.jsonl \"s3://$EVAL_BUCKET/input/dataset.jsonl\"\n```\n\nTen pairs is a hands-on size. The mechanics are identical at 500 — for a real project, this file *is* the asset worth investing in: it becomes your regression test for every future model release.\n\nBedrock runs the evaluation on your behalf, so it needs a role it can assume, with read/write on the bucket and permission to invoke models:\n\n```\ncat > /tmp/eval-trust.json << 'EOF'\n{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"bedrock.amazonaws.com\"},\"Action\":\"sts:AssumeRole\"}]}\nEOF\n\nEVAL_ROLE_ARN=$(aws iam create-role \\\n  --role-name BedrockEvaluationRole \\\n  --assume-role-policy-document file:///tmp/eval-trust.json \\\n  --query 'Role.Arn' --output text)\n\ncat > /tmp/eval-policy.json << EOF\n{\n  \"Version\": \"2012-10-17\",\n  \"Statement\": [\n    {\"Effect\":\"Allow\",\"Action\":[\"s3:GetObject\",\"s3:ListBucket\",\"s3:PutObject\"],\n     \"Resource\":[\"arn:aws:s3:::$EVAL_BUCKET\",\"arn:aws:s3:::$EVAL_BUCKET/*\"]},\n    {\"Effect\":\"Allow\",\"Action\":[\"bedrock:InvokeModel\"],\n     \"Resource\":[\"arn:aws:bedrock:*::foundation-model/*\",\"arn:aws:bedrock:*:*:inference-profile/*\"]}\n  ]\n}\nEOF\n\naws iam put-role-policy --role-name BedrockEvaluationRole \\\n  --policy-name inline --policy-document file:///tmp/eval-policy.json\n```\n\nNote the trust policy: the principal is `bedrock.amazonaws.com`\n\n— this role is for the service, not for you or a Lambda.\n\nIn the Bedrock console (Inference and Assessment → Evaluations), create two jobs against the same dataset — the video above walks through every screen:\n\n`s3://$EVAL_BUCKET/input/dataset.jsonl`\n\n, the `BedrockEvaluationRole`\n\n, and an output path under `s3://$EVAL_BUCKET/output/auto/`\n\n`s3://$EVAL_BUCKET/output/judge/`\n\nBoth jobs run asynchronously — expect several minutes to tens of minutes depending on dataset size.\n\nWhen the jobs complete, sync everything down and find the result files:\n\n```\naws s3 sync \"s3://$EVAL_BUCKET/output/\" /tmp/eval-out/\nfind /tmp/eval-out/ -type f\n\n# List the main output files\nfind /tmp/eval-out/ -name \"*_output.jsonl\" -not -name \"*.out\"\n```\n\nThe output is deeply nested JSONL — one record per prompt, each carrying a `scores`\n\narray. Aggregate the automatic metrics into per-metric count / mean / max:\n\n```\nfor f in /tmp/eval-out/auto/*/*/models/*/taskTypes/*/datasets/*/*_output.jsonl; do\n  cat \"$f\" | jq -r '.automatedEvaluationResult.scores[] | \"\\(.metricName)\\t\\(.result)\"'\ndone | awk -F'\\t' '\n  {sum[$1]+=$2; cnt[$1]++; if($2>max[$1]) max[$1]=$2}\n  END {for (m in sum) printf \"%s\\tcount %d\\tmean %.4f\\tmax %.4f\\n\", m, cnt[m], sum[m]/cnt[m], max[m]}\n'\n```\n\nAnd the judge's scores, averaged per metric:\n\n```\nJUDGE=$(find /tmp/eval-out/judge -name \"*_output.jsonl\" -not -name \"*.out\" | head -1)\n\ncat \"$JUDGE\" | jq -r '.automatedEvaluationResult.scores[] | \"\\(.metricName)\\t\\(.result)\"' | \\\n  awk '{sum[$1]+=$2; cnt[$1]++} END {for (m in sum) printf \"%s\\tmean %.2f\\n\", m, sum[m]/cnt[m]}'\n```\n\nTwo numbers-reading tips:\n\n```\n# S3\naws s3 rm \"s3://$EVAL_BUCKET\" --recursive\naws s3 rb \"s3://$EVAL_BUCKET\"\n\n# IAM\naws iam delete-role-policy --role-name BedrockEvaluationRole --policy-name inline\naws iam delete-role --role-name BedrockEvaluationRole\n```\n\nModel selection without evaluation is a vibe, and vibes don't survive the pace at which Bedrock ships new models. A ten-line JSONL file, one IAM role, and two evaluation jobs give you a repeatable benchmark you can rerun against every new release — and the same dataset doubles as a regression test when you change prompts, parameters, or routing. If you've been choosing models by eyeballing outputs, this is the upgrade.\n\n**About the author**\n\nMaruchin Tech — 12x AWS Certified | Cloud & AI for manufacturing and supply chain (AWS / Google Cloud / Azure) | Udemy instructor (100K+ students)\n\n🎥 Video version of this hands-on:\n\n[https://youtu.be/jbrFKA34hWc](https://youtu.be/jbrFKA34hWc)\n\n📚 Full course — AWS Certified Generative AI Developer Professional (AIP-C01) Exam Prep:\n\n[https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/](https://www.udemy.com/course/aws-certified-generative-ai-developer-professional-exam-prep/)\n\n👨🏫 All my courses:\n\n(Eng) [https://www.udemy.com/user/maruchin-tech-2/](https://www.udemy.com/user/maruchin-tech-2/)\n\n(Jpn) [https://www.udemy.com/user/shan-wang-wan-jun-2/](https://www.udemy.com/user/shan-wang-wan-jun-2/)\n\n🎫 Monthly discount coupons:\n\n[https://www.youtube.com/@MaruchinTech-cloud/posts](https://www.youtube.com/@MaruchinTech-cloud/posts)", "url": "https://wpnews.pro/news/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on", "canonical_source": "https://dev.to/maruchin_tech_555/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on-253k", "published_at": "2026-08-22 08:17:00+00:00", "updated_at": "2026-08-22 08:43:46.869747+00:00", "lang": "en", "topics": ["machine-learning", "large-language-models", "ai-products", "ai-tools", "mlops"], "entities": ["Amazon Bedrock", "AWS Lambda", "Amazon S3", "Amazon DynamoDB", "CloudWatch", "IAM", "Amazon VPC", "Amazon RDS"], "alternates": {"html": "https://wpnews.pro/news/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on", "markdown": "https://wpnews.pro/news/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on.md", "text": "https://wpnews.pro/news/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on.txt", "jsonld": "https://wpnews.pro/news/stop-guessing-which-model-is-better-amazon-bedrock-model-evaluation-hands-on.jsonld"}}