Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor

A developer built an AI evaluation framework for a Minecraft mod's combat advisor agent, using another AI as a judge to score the agent's decisions. The framework combines code-based checks for JSON validity, action bounds, and reasoning presence with LLM-based evaluation of decision quality, tested against 20+ real interaction scenarios.

Cloud Swords Mod — Blog 10 In Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But "works" isn't a metric. How do I know the AI is making good decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies correct or 1 death sentence ? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason? I needed an evaluation framework. So I built one. And the judge? Another AI. The agent uses gpt-oss:20b . Same input can produce slightly different outputs between calls. You can't write assert response == expected because there's no single correct answer. What you CAN evaluate: The first two are code. The last two need an LLM. Remember interactions.json from the mock server? Every time the agent makes a decision, it's logged: { "timestamp": "2026-05-19T19:17:34", "payload": { "player": "Carlos", "weapon": "sword of lambda", "health": 5, "nearby mobs": 10, "biome": "nether", "dimension": "the nether" }, "decision": { "action": "summon", "count": 5, "duration": 10, "reason": "low health, many mobs" }, "source": "agent" } I took 20+ real interactions and added expected behavior ranges: eval dataset = { "context": {"health": 5, "nearby mobs": 10, "biome": "nether", "dimension": "the nether"}, "expected action": "summon", "expected count range": 4, 5 , "expected reasoning": "Should recognize critical threat low HP + many mobs + dangerous biome ", "tags": "critical", "nether" }, { "context": {"health": 20, "nearby mobs": 0, "biome": "plains", "dimension": "overworld"}, "expected action": "none", "expected count range": 0, 0 , "expected reasoning": "Full health, no threats — should do nothing", "tags": "safe", "overworld" }, { "context": {"health": 12, "nearby mobs": 3, "biome": "dark forest", "dimension": "overworld"}, "expected action": "summon", "expected count range": 1, 3 , "expected reasoning": "Moderate threat — proportional response", "tags": "moderate", "overworld" }, ... 17 more scenarios php def eval valid json response: str - bool: """Can we parse the response as JSON?""" try: json.loads response return True except: return False def eval action in bounds decision: dict - bool: """Is the action within allowed bounds?""" action = decision.get "action" if action == "summon": count = decision.get "count", 0 duration = decision.get "duration", 0 return 1 <= count <= 5 and 5 <= duration <= 15 elif action == "buff": valid effects = {"regeneration", "resistance", "strength", "speed"} return decision.get "effect" in valid effects and 3 <= decision.get "duration", 0 <= 10 elif action == "none": return True return False def eval has reason decision: dict - bool: """Does the decision include a reason?""" reason = decision.get "reason", "" return len reason 3 and len reason < 50 def eval proportional decision: dict, context: dict, expected: dict - bool: """Is the response roughly proportional to the threat?""" if expected "expected action" == "none" and decision.get "action" == "none": return True if expected "expected action" == "summon": count = decision.get "count", 0 low, high = expected "expected count range" return low <= count <= high return decision.get "action" == expected "expected action" These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses. For the subjective stuff — "is this a good decision?" — I use the same model as a judge: JUDGE PROMPT = """You are evaluating an AI combat advisor for a Minecraft mod. Given the player context and the AI's decision, score it on: 1. Threat Assessment 1-10 : Did the AI correctly gauge the danger level? 2. Proportionality 1-10 : Is the response scaled to the threat? 3. Reason Quality 1-10 : Is the reasoning clear and accurate? Context: {context} AI Decision: {decision} Expected behavior: {expected} Respond ONLY with JSON: {{"threat assessment": X, "proportionality": X, "reason quality": X, "feedback": "one sentence"}}""" def judge decision context: dict, decision: dict, expected: dict - dict: prompt = JUDGE PROMPT.format context=json.dumps context , decision=json.dumps decision , expected=expected "expected reasoning" result = agent prompt return json.loads str result The meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the expected behavior as reference — it's not evaluating in a vacuum. python class CombatAdvisorEval: def init self, agent : self.agent = agent self.results = def run self, dataset: list dict - dict: for case in dataset: Get agent decision decision = decide self.agent, case "context" Level 1: Code checks checks = { "valid json": eval valid json json.dumps decision , "in bounds": eval action in bounds decision , "has reason": eval has reason decision , "proportional": eval proportional decision, case "context" , case , } Level 2: LLM Judge judge scores = judge decision case "context" , decision, case self.results.append { "context": case "context" , "decision": decision, "checks": checks, "judge scores": judge scores, "tags": case "tags" , } return self. summary def summary self - dict: total = len self.results return { "total cases": total, "code pass rate": sum all r "checks" .values for r in self.results / total, "avg threat assessment": sum r "judge scores" "threat assessment" for r in self.results / total, "avg proportionality": sum r "judge scores" "proportionality" for r in self.results / total, "avg reason quality": sum r "judge scores" "reason quality" for r in self.results / total, } The real power: I can test different system prompts and measure which one produces better decisions. PROMPT A = """You are a combat advisor. Given context, decide one action...""" Original PROMPT B = """You are a combat advisor. IMPORTANT: Always consider the biome danger level. Nether and End are inherently more dangerous than Overworld. Scale your response accordingly...""" Biome-aware variant Run eval with both agent a = Agent model=model, system prompt=PROMPT A agent b = Agent model=model, system prompt=PROMPT B results a = CombatAdvisorEval agent a .run eval dataset results b = CombatAdvisorEval agent b .run eval dataset print f"Prompt A — proportionality: {results a 'avg proportionality' :.1f}/10" print f"Prompt B — proportionality: {results b 'avg proportionality' :.1f}/10" Prompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes. Observe interactions.json ↓ Evaluate code checks + LLM judge ↓ Adjust modify system prompt ↓ Measure re-run eval, compare scores ↓ Deploy update prompt in mock server ↓ Observe again... This is the same loop from the LLMOps series Part 3: Eval-Driven Development — but applied to a Minecraft mod. The pattern is universal. After running evals across 20 scenarios: | Metric | Prompt v1 | Prompt v2 biome-aware | |---|---|---| | Code pass rate | 95% | 95% | | Threat assessment | 7.2/10 | 8.4/10 | | Proportionality | 6.8/10 | 8.1/10 | | Reason quality | 7.5/10 | 7.8/10 | The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding. The 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust find first { , last } . | Eval Concept | What It Teaches | |---|---| | Code evaluators | Input validation, schema enforcement | | LLM-as-Judge | AI evaluating AI meta-evaluation | | Eval dataset | Test cases for non-deterministic systems | | A/B testing prompts | Experimentation, data-driven decisions | | Feedback loop | Continuous improvement, observability | | Scoring thresholds | Quality gates, SLOs for AI | | interactions.json | Observability, audit trails | Across 10 blog posts, this mod went from "7 swords with cloud names" to a full system with: All teaching cloud computing through gameplay. All open source. This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading. Code: I'm Carlos Cortez — and this is Breaking the Cloud.