Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor

Cloud Swords Mod — Blog 10

In Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But "works" isn't a metric.

How do I know the AI is making good decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?

I needed an evaluation framework. So I built one. And the judge? Another AI.

The agent uses gpt-oss:20b

. Same input can produce slightly different outputs between calls. You can't write assert response == expected

because there's no single correct answer.

What you CAN evaluate:

The first two are code. The last two need an LLM.

Remember interactions.json

from the mock server? Every time the agent makes a decision, it's logged:

{
  "timestamp": "2026-05-19T19:17:34",
  "payload": {
    "player": "Carlos",
    "weapon": "sword_of_lambda",
    "health": 5,
    "nearby_mobs": 10,
    "biome": "nether",
    "dimension": "the_nether"
  },
  "decision": {
    "action": "summon",
    "count": 5,
    "duration": 10,
    "reason": "low health, many mobs"
  },
  "source": "agent"
}

I took 20+ real interactions and added expected behavior ranges:

eval_dataset = [
    {
        "context": {"health": 5, "nearby_mobs": 10, "biome": "nether", "dimension": "the_nether"},
        "expected_action": "summon",
        "expected_count_range": [4, 5],
        "expected_reasoning": "Should recognize critical threat (low HP + many mobs + dangerous biome)",
        "tags": ["critical", "nether"]
    },
    {
        "context": {"health": 20, "nearby_mobs": 0, "biome": "plains", "dimension": "overworld"},
        "expected_action": "none",
        "expected_count_range": [0, 0],
        "expected_reasoning": "Full health, no threats — should do nothing",
        "tags": ["safe", "overworld"]
    },
    {
        "context": {"health": 12, "nearby_mobs": 3, "biome": "dark_forest", "dimension": "overworld"},
        "expected_action": "summon",
        "expected_count_range": [1, 3],
        "expected_reasoning": "Moderate threat — proportional response",
        "tags": ["moderate", "overworld"]
    },
]
php
def eval_valid_json(response: str) -> bool:
    """Can we parse the response as JSON?"""
    try:
        json.loads(response)
        return True
    except:
        return False

def eval_action_in_bounds(decision: dict) -> bool:
    """Is the action within allowed bounds?"""
    action = decision.get("action")
    if action == "summon":
        count = decision.get("count", 0)
        duration = decision.get("duration", 0)
        return 1 <= count <= 5 and 5 <= duration <= 15
    elif action == "buff":
        valid_effects = {"regeneration", "resistance", "strength", "speed"}
        return decision.get("effect") in valid_effects and 3 <= decision.get("duration", 0) <= 10
    elif action == "none":
        return True
    return False

def eval_has_reason(decision: dict) -> bool:
    """Does the decision include a reason?"""
    reason = decision.get("reason", "")
    return len(reason) > 3 and len(reason) < 50

def eval_proportional(decision: dict, context: dict, expected: dict) -> bool:
    """Is the response roughly proportional to the threat?"""
    if expected["expected_action"] == "none" and decision.get("action") == "none":
        return True
    if expected["expected_action"] == "summon":
        count = decision.get("count", 0)
        low, high = expected["expected_count_range"]
        return low <= count <= high
    return decision.get("action") == expected["expected_action"]

These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.

For the subjective stuff — "is this a good decision?" — I use the same model as a judge:

JUDGE_PROMPT = """You are evaluating an AI combat advisor for a Minecraft mod.

Given the player context and the AI's decision, score it on:
1. **Threat Assessment** (1-10): Did the AI correctly gauge the danger level?
2. **Proportionality** (1-10): Is the response scaled to the threat?
3. **Reason Quality** (1-10): Is the reasoning clear and accurate?

Context: {context}
AI Decision: {decision}
Expected behavior: {expected}

Respond ONLY with JSON:
{{"threat_assessment": X, "proportionality": X, "reason_quality": X, "feedback": "one sentence"}}"""

def judge_decision(context: dict, decision: dict, expected: dict) -> dict:
    prompt = JUDGE_PROMPT.format(
        context=json.dumps(context),
        decision=json.dumps(decision),
        expected=expected["expected_reasoning"]
    )
    result = agent(prompt)
    return json.loads(str(result))

The meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the expected behavior as reference — it's not evaluating in a vacuum.

class CombatAdvisorEval:
    def __init__(self, agent):
        self.agent = agent
        self.results = []

    def run(self, dataset: list[dict]) -> dict:
        for case in dataset:
            decision = decide(self.agent, case["context"])

            checks = {
                "valid_json": eval_valid_json(json.dumps(decision)),
                "in_bounds": eval_action_in_bounds(decision),
                "has_reason": eval_has_reason(decision),
                "proportional": eval_proportional(decision, case["context"], case),
            }

            judge_scores = judge_decision(case["context"], decision, case)

            self.results.append({
                "context": case["context"],
                "decision": decision,
                "checks": checks,
                "judge_scores": judge_scores,
                "tags": case["tags"],
            })

        return self._summary()

    def _summary(self) -> dict:
        total = len(self.results)
        return {
            "total_cases": total,
            "code_pass_rate": sum(
                all(r["checks"].values()) for r in self.results
            ) / total,
            "avg_threat_assessment": sum(
                r["judge_scores"]["threat_assessment"] for r in self.results
            ) / total,
            "avg_proportionality": sum(
                r["judge_scores"]["proportionality"] for r in self.results
            ) / total,
            "avg_reason_quality": sum(
                r["judge_scores"]["reason_quality"] for r in self.results
            ) / total,
        }

The real power: I can test different system prompts and measure which one produces better decisions.

PROMPT_A = """You are a combat advisor. Given context, decide one action..."""  # Original

PROMPT_B = """You are a combat advisor. IMPORTANT: Always consider the biome danger level.
Nether and End are inherently more dangerous than Overworld.
Scale your response accordingly..."""  # Biome-aware variant

agent_a = Agent(model=model, system_prompt=PROMPT_A)
agent_b = Agent(model=model, system_prompt=PROMPT_B)

results_a = CombatAdvisorEval(agent_a).run(eval_dataset)
results_b = CombatAdvisorEval(agent_b).run(eval_dataset)

print(f"Prompt A — proportionality: {results_a['avg_proportionality']:.1f}/10")
print(f"Prompt B — proportionality: {results_b['avg_proportionality']:.1f}/10")

Prompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes.

Observe (interactions.json)
    ↓
Evaluate (code checks + LLM judge)
    ↓
Adjust (modify system prompt)
    ↓
Measure (re-run eval, compare scores)
    ↓
Deploy (update prompt in mock server)
    ↓
Observe again...

This is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod. The pattern is universal.

After running evals across 20 scenarios:

The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.

The 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust (find first {

, last }

).

Across 10 blog posts, this mod went from "7 swords with cloud names" to a full system with:

All teaching cloud computing through gameplay. All open source.

This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading.

Code:

I'm Carlos Cortez — and this is Breaking the Cloud.