{"slug": "an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject", "title": "An AWS Labs agent-eval sample uses the same model as judge and subject", "summary": "An AWS Labs sample for evaluating AI agents, Agent-EvalKit, uses the same Anthropic Claude model as both the judge and the subject in its QA example, a design choice that is not disclosed in the repository. The developer who discovered this notes that the judge model defaults to the agent's model without any configuration or documentation, highlighting a common failure mode in evaluation harnesses where self-grading is hidden as a default.", "body_md": "`awslabs/Agent-EvalKit`\n\nis a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in `examples/qa_agent_evaluation/eval/metrics.py`\n\n:\n\n```\nclass BedrockLLM(DeepEvalBaseLLM):\n    \"\"\"Custom DeepEval LLM using LiteLLM to call Bedrock.\"\"\"\n\n    def __init__(self, model: str = \"bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0\"):\n```\n\nBoth metric factories instantiate it with no argument:\n\n``` php\ndef create_faithfulness_metric(threshold: float = 0.7) -> FaithfulnessMetric:\n    bedrock_llm = BedrockLLM()\n    return FaithfulnessMetric(threshold=threshold, model=bedrock_llm, include_reason=True)\n```\n\nThe agent under evaluation runs `us.anthropic.claude-sonnet-4-20250514-v1:0`\n\n— that is `gen_ai.request.model`\n\nin the example's own `eval/otel-traces.jsonl`\n\n, and it is the only model id that appears anywhere in that trace file.\n\nSo the judge and the subject are the same model. (Not the same string: the judge carries LiteLLM's `bedrock/`\n\nroute prefix. Same model, one hop of indirection.) The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured.\n\nCheck it yourself:\n\n```\ncurl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \\\n  && cd awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval \\\n  && echo \"judge:\"   && grep -o 'bedrock/us\\.anthropic\\.[a-z0-9.:-]*' metrics.py | sort -u \\\n  && echo \"subject:\" && grep -o 'us\\.anthropic\\.[a-z0-9.:-]*' otel-traces.jsonl | sort -u \\\n  && echo \"caveats:\" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \\\n       ../../../ --include=*.md --include=*.py ; echo \"^ nothing under 'caveats:' = undisclosed\"\n```\n\nConfirmed at `472fbc6`\n\n.\n\nNot \"AWS shipped something bad.\" It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post.\n\nThe point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all.\n\nThat is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads.\n\nAnd `examples/`\n\nis precisely where defaults go to breed. Sample directories exist to be copied.\n\nI build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command.\n\nAnd the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — \"clean\" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work.\n\nI also filed an unrelated bug in the same file — a debug `exit()`\n\nthat stops the example running — as [issue #4](https://github.com/awslabs/Agent-EvalKit/issues/4), with a fix in [PR #5](https://github.com/awslabs/Agent-EvalKit/pull/5).", "url": "https://wpnews.pro/news/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject", "canonical_source": "https://dev.to/michael_hurst_c009b1bdeb8/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject-e29", "published_at": "2026-08-31 19:59:48+00:00", "updated_at": "2026-08-31 20:24:17.297720+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools"], "entities": ["AWS Labs", "Agent-EvalKit", "Anthropic", "Claude", "DeepEval", "LiteLLM", "Bedrock"], "alternates": {"html": "https://wpnews.pro/news/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject", "markdown": "https://wpnews.pro/news/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject.md", "text": "https://wpnews.pro/news/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject.txt", "jsonld": "https://wpnews.pro/news/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject.jsonld"}}