# Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor

> Source: <https://dev.to/ccortezb/-blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores-to-make-it-a-333j>
> Published: 2026-08-01 16:13:45+00:00

*Cloud Swords Mod — Blog 10*

In Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But "works" isn't a metric.

How do I know the AI is making *good* decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?

I needed an evaluation framework. So I built one. And the judge? Another AI.

The agent uses `gpt-oss:20b`

. Same input can produce slightly different outputs between calls. You can't write `assert response == expected`

because there's no single correct answer.

What you CAN evaluate:

The first two are code. The last two need an LLM.

Remember `interactions.json`

from the mock server? Every time the agent makes a decision, it's logged:

```
{
  "timestamp": "2026-05-19T19:17:34",
  "payload": {
    "player": "Carlos",
    "weapon": "sword_of_lambda",
    "health": 5,
    "nearby_mobs": 10,
    "biome": "nether",
    "dimension": "the_nether"
  },
  "decision": {
    "action": "summon",
    "count": 5,
    "duration": 10,
    "reason": "low health, many mobs"
  },
  "source": "agent"
}
```

I took 20+ real interactions and added expected behavior ranges:

```
eval_dataset = [
    {
        "context": {"health": 5, "nearby_mobs": 10, "biome": "nether", "dimension": "the_nether"},
        "expected_action": "summon",
        "expected_count_range": [4, 5],
        "expected_reasoning": "Should recognize critical threat (low HP + many mobs + dangerous biome)",
        "tags": ["critical", "nether"]
    },
    {
        "context": {"health": 20, "nearby_mobs": 0, "biome": "plains", "dimension": "overworld"},
        "expected_action": "none",
        "expected_count_range": [0, 0],
        "expected_reasoning": "Full health, no threats — should do nothing",
        "tags": ["safe", "overworld"]
    },
    {
        "context": {"health": 12, "nearby_mobs": 3, "biome": "dark_forest", "dimension": "overworld"},
        "expected_action": "summon",
        "expected_count_range": [1, 3],
        "expected_reasoning": "Moderate threat — proportional response",
        "tags": ["moderate", "overworld"]
    },
    # ... 17 more scenarios
]
php
def eval_valid_json(response: str) -> bool:
    """Can we parse the response as JSON?"""
    try:
        json.loads(response)
        return True
    except:
        return False

def eval_action_in_bounds(decision: dict) -> bool:
    """Is the action within allowed bounds?"""
    action = decision.get("action")
    if action == "summon":
        count = decision.get("count", 0)
        duration = decision.get("duration", 0)
        return 1 <= count <= 5 and 5 <= duration <= 15
    elif action == "buff":
        valid_effects = {"regeneration", "resistance", "strength", "speed"}
        return decision.get("effect") in valid_effects and 3 <= decision.get("duration", 0) <= 10
    elif action == "none":
        return True
    return False

def eval_has_reason(decision: dict) -> bool:
    """Does the decision include a reason?"""
    reason = decision.get("reason", "")
    return len(reason) > 3 and len(reason) < 50

def eval_proportional(decision: dict, context: dict, expected: dict) -> bool:
    """Is the response roughly proportional to the threat?"""
    if expected["expected_action"] == "none" and decision.get("action") == "none":
        return True
    if expected["expected_action"] == "summon":
        count = decision.get("count", 0)
        low, high = expected["expected_count_range"]
        return low <= count <= high
    return decision.get("action") == expected["expected_action"]
```

These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.

For the subjective stuff — "is this a good decision?" — I use the same model as a judge:

```
JUDGE_PROMPT = """You are evaluating an AI combat advisor for a Minecraft mod.

Given the player context and the AI's decision, score it on:
1. **Threat Assessment** (1-10): Did the AI correctly gauge the danger level?
2. **Proportionality** (1-10): Is the response scaled to the threat?
3. **Reason Quality** (1-10): Is the reasoning clear and accurate?

Context: {context}
AI Decision: {decision}
Expected behavior: {expected}

Respond ONLY with JSON:
{{"threat_assessment": X, "proportionality": X, "reason_quality": X, "feedback": "one sentence"}}"""

def judge_decision(context: dict, decision: dict, expected: dict) -> dict:
    prompt = JUDGE_PROMPT.format(
        context=json.dumps(context),
        decision=json.dumps(decision),
        expected=expected["expected_reasoning"]
    )
    result = agent(prompt)
    return json.loads(str(result))
```

The meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the *expected behavior* as reference — it's not evaluating in a vacuum.

``` python
class CombatAdvisorEval:
    def __init__(self, agent):
        self.agent = agent
        self.results = []

    def run(self, dataset: list[dict]) -> dict:
        for case in dataset:
            # Get agent decision
            decision = decide(self.agent, case["context"])

            # Level 1: Code checks
            checks = {
                "valid_json": eval_valid_json(json.dumps(decision)),
                "in_bounds": eval_action_in_bounds(decision),
                "has_reason": eval_has_reason(decision),
                "proportional": eval_proportional(decision, case["context"], case),
            }

            # Level 2: LLM Judge
            judge_scores = judge_decision(case["context"], decision, case)

            self.results.append({
                "context": case["context"],
                "decision": decision,
                "checks": checks,
                "judge_scores": judge_scores,
                "tags": case["tags"],
            })

        return self._summary()

    def _summary(self) -> dict:
        total = len(self.results)
        return {
            "total_cases": total,
            "code_pass_rate": sum(
                all(r["checks"].values()) for r in self.results
            ) / total,
            "avg_threat_assessment": sum(
                r["judge_scores"]["threat_assessment"] for r in self.results
            ) / total,
            "avg_proportionality": sum(
                r["judge_scores"]["proportionality"] for r in self.results
            ) / total,
            "avg_reason_quality": sum(
                r["judge_scores"]["reason_quality"] for r in self.results
            ) / total,
        }
```

The real power: I can test different system prompts and measure which one produces better decisions.

```
PROMPT_A = """You are a combat advisor. Given context, decide one action..."""  # Original

PROMPT_B = """You are a combat advisor. IMPORTANT: Always consider the biome danger level.
Nether and End are inherently more dangerous than Overworld.
Scale your response accordingly..."""  # Biome-aware variant

# Run eval with both
agent_a = Agent(model=model, system_prompt=PROMPT_A)
agent_b = Agent(model=model, system_prompt=PROMPT_B)

results_a = CombatAdvisorEval(agent_a).run(eval_dataset)
results_b = CombatAdvisorEval(agent_b).run(eval_dataset)

print(f"Prompt A — proportionality: {results_a['avg_proportionality']:.1f}/10")
print(f"Prompt B — proportionality: {results_b['avg_proportionality']:.1f}/10")
```

Prompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes.

```
Observe (interactions.json)
    ↓
Evaluate (code checks + LLM judge)
    ↓
Adjust (modify system prompt)
    ↓
Measure (re-run eval, compare scores)
    ↓
Deploy (update prompt in mock server)
    ↓
Observe again...
```

This is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod. The pattern is universal.

After running evals across 20 scenarios:

| Metric | Prompt v1 | Prompt v2 (biome-aware) |
|---|---|---|
| Code pass rate | 95% | 95% |
| Threat assessment | 7.2/10 | 8.4/10 |
| Proportionality | 6.8/10 | 8.1/10 |
| Reason quality | 7.5/10 | 7.8/10 |

The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.

The 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust (find first `{`

, last `}`

).

| Eval Concept | What It Teaches |
|---|---|
| Code evaluators | Input validation, schema enforcement |
| LLM-as-Judge | AI evaluating AI (meta-evaluation) |
| Eval dataset | Test cases for non-deterministic systems |
| A/B testing prompts | Experimentation, data-driven decisions |
| Feedback loop | Continuous improvement, observability |
| Scoring thresholds | Quality gates, SLOs for AI |
| interactions.json | Observability, audit trails |

Across 10 blog posts, this mod went from "7 swords with cloud names" to a full system with:

All teaching cloud computing through gameplay. All open source.

*This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading.*

**Code:**

*I'm Carlos Cortez — and this is Breaking the Cloud.*