An AWS Labs agent-eval sample uses the same model as judge and subject

An AWS Labs sample for evaluating AI agents, Agent-EvalKit, uses the same Anthropic Claude model as both the judge and the subject in its QA example, a design choice that is not disclosed in the repository. The developer who discovered this notes that the judge model defaults to the agent's model without any configuration or documentation, highlighting a common failure mode in evaluation harnesses where self-grading is hidden as a default.

awslabs/Agent-EvalKit is a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in examples/qa agent evaluation/eval/metrics.py : class BedrockLLM DeepEvalBaseLLM : """Custom DeepEval LLM using LiteLLM to call Bedrock.""" def init self, model: str = "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0" : Both metric factories instantiate it with no argument: php def create faithfulness metric threshold: float = 0.7 - FaithfulnessMetric: bedrock llm = BedrockLLM return FaithfulnessMetric threshold=threshold, model=bedrock llm, include reason=True The agent under evaluation runs us.anthropic.claude-sonnet-4-20250514-v1:0 — that is gen ai.request.model in the example's own eval/otel-traces.jsonl , and it is the only model id that appears anywhere in that trace file. So the judge and the subject are the same model. Not the same string: the judge carries LiteLLM's bedrock/ route prefix. Same model, one hop of indirection. The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured. Check it yourself: curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \ && cd awslabs-Agent-EvalKit- /examples/qa agent evaluation/eval \ && echo "judge:" && grep -o 'bedrock/us\.anthropic\. a-z0-9.:- ' metrics.py | sort -u \ && echo "subject:" && grep -o 'us\.anthropic\. a-z0-9.:- ' otel-traces.jsonl | sort -u \ && echo "caveats:" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \ ../../../ --include= .md --include= .py ; echo "^ nothing under 'caveats:' = undisclosed" Confirmed at 472fbc6 . Not "AWS shipped something bad." It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post. The point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all. That is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads. And examples/ is precisely where defaults go to breed. Sample directories exist to be copied. I build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command. And the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — "clean" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work. I also filed an unrelated bug in the same file — a debug exit that stops the example running — as issue 4 https://github.com/awslabs/Agent-EvalKit/issues/4 , with a fix in PR 5 https://github.com/awslabs/Agent-EvalKit/pull/5 .