# An AWS Labs agent-eval sample uses the same model as judge and subject

> Source: <https://dev.to/michael_hurst_c009b1bdeb8/an-aws-labs-agent-eval-sample-uses-the-same-model-as-judge-and-subject-e29>
> Published: 2026-08-31 19:59:48+00:00

`awslabs/Agent-EvalKit`

is a sample kit for evaluating AI agents. Its QA example defines the DeepEval judge in `examples/qa_agent_evaluation/eval/metrics.py`

:

```
class BedrockLLM(DeepEvalBaseLLM):
    """Custom DeepEval LLM using LiteLLM to call Bedrock."""

    def __init__(self, model: str = "bedrock/us.anthropic.claude-sonnet-4-20250514-v1:0"):
```

Both metric factories instantiate it with no argument:

``` php
def create_faithfulness_metric(threshold: float = 0.7) -> FaithfulnessMetric:
    bedrock_llm = BedrockLLM()
    return FaithfulnessMetric(threshold=threshold, model=bedrock_llm, include_reason=True)
```

The agent under evaluation runs `us.anthropic.claude-sonnet-4-20250514-v1:0`

— that is `gen_ai.request.model`

in the example's own `eval/otel-traces.jsonl`

, and it is the only model id that appears anywhere in that trace file.

So the judge and the subject are the same model. (Not the same string: the judge carries LiteLLM's `bedrock/`

route prefix. Same model, one hop of indirection.) The bundled report gives the agent a faithfulness score of 78.2%, and the thing measuring faithfulness is the thing whose faithfulness is being measured.

Check it yourself:

```
curl -sL https://api.github.com/repos/awslabs/Agent-EvalKit/tarball/main | tar xz \
  && cd awslabs-Agent-EvalKit-*/examples/qa_agent_evaluation/eval \
  && echo "judge:"   && grep -o 'bedrock/us\.anthropic\.[a-z0-9.:-]*' metrics.py | sort -u \
  && echo "subject:" && grep -o 'us\.anthropic\.[a-z0-9.:-]*' otel-traces.jsonl | sort -u \
  && echo "caveats:" && grep -rniE 'same model|self-grad|bias|independent|different model|cross-model|judge model' \
       ../../../ --include=*.md --include=*.py ; echo "^ nothing under 'caveats:' = undisclosed"
```

Confirmed at `472fbc6`

.

Not "AWS shipped something bad." It is a Labs sample — 38 stars at the time of writing, last pushed in March 2026 — and using one Bedrock model for both roles is defensible on cost, latency and dependency count. If someone had written that tradeoff down I would have no post.

The point is that nobody chose it. It is a kwarg default on a helper class. There is no config surface for the judge model, no line in the README noting that the judge defaults to whatever the agent runs on, and — per that last grep — no mention anywhere in the repo of judge independence, model bias, or cross-model evaluation. The most consequential decision in an evaluation harness was never surfaced as a decision at all.

That is the failure mode I think is worth naming. Self-grading almost never arrives as a claim you can disagree with. It arrives as a default, three call frames down, in a constructor nobody reads.

And `examples/`

is precisely where defaults go to breed. Sample directories exist to be copied.

I build an independent validation gate, so treat this as an interested party pointing at a thing that flatters him, and go run the command.

And the number that matters here is the unflattering one, so: on a 113-example internal holdout, our hosted API flagged 19.5% of the clean items — "clean" meaning an item our adjudicated labels recorded as containing no defect, so a flag on one is the gate being wrong, not the item. Banked 2026-08-10. That is our data on our holdout; yours will differ. A separate judge is not a correct judge. It is only a judge that is not marking its own work.

I also filed an unrelated bug in the same file — a debug `exit()`

that stops the example running — as [issue #4](https://github.com/awslabs/Agent-EvalKit/issues/4), with a fix in [PR #5](https://github.com/awslabs/Agent-EvalKit/pull/5).