{"slug": "blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores", "title": "Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor", "summary": "A developer built an AI evaluation framework for a Minecraft mod's combat advisor agent, using another AI as a judge to score the agent's decisions. The framework combines code-based checks for JSON validity, action bounds, and reasoning presence with LLM-based evaluation of decision quality, tested against 20+ real interaction scenarios.", "body_md": "*Cloud Swords Mod — Blog 10*\n\nIn Blog 9, I gave my Minecraft mod an AI brain. A Strands Agent that decides how many allies to summon based on your combat situation. It works. But \"works\" isn't a metric.\n\nHow do I know the AI is making *good* decisions? When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)? When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?\n\nI needed an evaluation framework. So I built one. And the judge? Another AI.\n\nThe agent uses `gpt-oss:20b`\n\n. Same input can produce slightly different outputs between calls. You can't write `assert response == expected`\n\nbecause there's no single correct answer.\n\nWhat you CAN evaluate:\n\nThe first two are code. The last two need an LLM.\n\nRemember `interactions.json`\n\nfrom the mock server? Every time the agent makes a decision, it's logged:\n\n```\n{\n  \"timestamp\": \"2026-05-19T19:17:34\",\n  \"payload\": {\n    \"player\": \"Carlos\",\n    \"weapon\": \"sword_of_lambda\",\n    \"health\": 5,\n    \"nearby_mobs\": 10,\n    \"biome\": \"nether\",\n    \"dimension\": \"the_nether\"\n  },\n  \"decision\": {\n    \"action\": \"summon\",\n    \"count\": 5,\n    \"duration\": 10,\n    \"reason\": \"low health, many mobs\"\n  },\n  \"source\": \"agent\"\n}\n```\n\nI took 20+ real interactions and added expected behavior ranges:\n\n```\neval_dataset = [\n    {\n        \"context\": {\"health\": 5, \"nearby_mobs\": 10, \"biome\": \"nether\", \"dimension\": \"the_nether\"},\n        \"expected_action\": \"summon\",\n        \"expected_count_range\": [4, 5],\n        \"expected_reasoning\": \"Should recognize critical threat (low HP + many mobs + dangerous biome)\",\n        \"tags\": [\"critical\", \"nether\"]\n    },\n    {\n        \"context\": {\"health\": 20, \"nearby_mobs\": 0, \"biome\": \"plains\", \"dimension\": \"overworld\"},\n        \"expected_action\": \"none\",\n        \"expected_count_range\": [0, 0],\n        \"expected_reasoning\": \"Full health, no threats — should do nothing\",\n        \"tags\": [\"safe\", \"overworld\"]\n    },\n    {\n        \"context\": {\"health\": 12, \"nearby_mobs\": 3, \"biome\": \"dark_forest\", \"dimension\": \"overworld\"},\n        \"expected_action\": \"summon\",\n        \"expected_count_range\": [1, 3],\n        \"expected_reasoning\": \"Moderate threat — proportional response\",\n        \"tags\": [\"moderate\", \"overworld\"]\n    },\n    # ... 17 more scenarios\n]\nphp\ndef eval_valid_json(response: str) -> bool:\n    \"\"\"Can we parse the response as JSON?\"\"\"\n    try:\n        json.loads(response)\n        return True\n    except:\n        return False\n\ndef eval_action_in_bounds(decision: dict) -> bool:\n    \"\"\"Is the action within allowed bounds?\"\"\"\n    action = decision.get(\"action\")\n    if action == \"summon\":\n        count = decision.get(\"count\", 0)\n        duration = decision.get(\"duration\", 0)\n        return 1 <= count <= 5 and 5 <= duration <= 15\n    elif action == \"buff\":\n        valid_effects = {\"regeneration\", \"resistance\", \"strength\", \"speed\"}\n        return decision.get(\"effect\") in valid_effects and 3 <= decision.get(\"duration\", 0) <= 10\n    elif action == \"none\":\n        return True\n    return False\n\ndef eval_has_reason(decision: dict) -> bool:\n    \"\"\"Does the decision include a reason?\"\"\"\n    reason = decision.get(\"reason\", \"\")\n    return len(reason) > 3 and len(reason) < 50\n\ndef eval_proportional(decision: dict, context: dict, expected: dict) -> bool:\n    \"\"\"Is the response roughly proportional to the threat?\"\"\"\n    if expected[\"expected_action\"] == \"none\" and decision.get(\"action\") == \"none\":\n        return True\n    if expected[\"expected_action\"] == \"summon\":\n        count = decision.get(\"count\", 0)\n        low, high = expected[\"expected_count_range\"]\n        return low <= count <= high\n    return decision.get(\"action\") == expected[\"expected_action\"]\n```\n\nThese catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.\n\nFor the subjective stuff — \"is this a good decision?\" — I use the same model as a judge:\n\n```\nJUDGE_PROMPT = \"\"\"You are evaluating an AI combat advisor for a Minecraft mod.\n\nGiven the player context and the AI's decision, score it on:\n1. **Threat Assessment** (1-10): Did the AI correctly gauge the danger level?\n2. **Proportionality** (1-10): Is the response scaled to the threat?\n3. **Reason Quality** (1-10): Is the reasoning clear and accurate?\n\nContext: {context}\nAI Decision: {decision}\nExpected behavior: {expected}\n\nRespond ONLY with JSON:\n{{\"threat_assessment\": X, \"proportionality\": X, \"reason_quality\": X, \"feedback\": \"one sentence\"}}\"\"\"\n\ndef judge_decision(context: dict, decision: dict, expected: dict) -> dict:\n    prompt = JUDGE_PROMPT.format(\n        context=json.dumps(context),\n        decision=json.dumps(decision),\n        expected=expected[\"expected_reasoning\"]\n    )\n    result = agent(prompt)\n    return json.loads(str(result))\n```\n\nThe meta part: the same AI model that makes the decisions is also judging them. This works because the judge has the *expected behavior* as reference — it's not evaluating in a vacuum.\n\n``` python\nclass CombatAdvisorEval:\n    def __init__(self, agent):\n        self.agent = agent\n        self.results = []\n\n    def run(self, dataset: list[dict]) -> dict:\n        for case in dataset:\n            # Get agent decision\n            decision = decide(self.agent, case[\"context\"])\n\n            # Level 1: Code checks\n            checks = {\n                \"valid_json\": eval_valid_json(json.dumps(decision)),\n                \"in_bounds\": eval_action_in_bounds(decision),\n                \"has_reason\": eval_has_reason(decision),\n                \"proportional\": eval_proportional(decision, case[\"context\"], case),\n            }\n\n            # Level 2: LLM Judge\n            judge_scores = judge_decision(case[\"context\"], decision, case)\n\n            self.results.append({\n                \"context\": case[\"context\"],\n                \"decision\": decision,\n                \"checks\": checks,\n                \"judge_scores\": judge_scores,\n                \"tags\": case[\"tags\"],\n            })\n\n        return self._summary()\n\n    def _summary(self) -> dict:\n        total = len(self.results)\n        return {\n            \"total_cases\": total,\n            \"code_pass_rate\": sum(\n                all(r[\"checks\"].values()) for r in self.results\n            ) / total,\n            \"avg_threat_assessment\": sum(\n                r[\"judge_scores\"][\"threat_assessment\"] for r in self.results\n            ) / total,\n            \"avg_proportionality\": sum(\n                r[\"judge_scores\"][\"proportionality\"] for r in self.results\n            ) / total,\n            \"avg_reason_quality\": sum(\n                r[\"judge_scores\"][\"reason_quality\"] for r in self.results\n            ) / total,\n        }\n```\n\nThe real power: I can test different system prompts and measure which one produces better decisions.\n\n```\nPROMPT_A = \"\"\"You are a combat advisor. Given context, decide one action...\"\"\"  # Original\n\nPROMPT_B = \"\"\"You are a combat advisor. IMPORTANT: Always consider the biome danger level.\nNether and End are inherently more dangerous than Overworld.\nScale your response accordingly...\"\"\"  # Biome-aware variant\n\n# Run eval with both\nagent_a = Agent(model=model, system_prompt=PROMPT_A)\nagent_b = Agent(model=model, system_prompt=PROMPT_B)\n\nresults_a = CombatAdvisorEval(agent_a).run(eval_dataset)\nresults_b = CombatAdvisorEval(agent_b).run(eval_dataset)\n\nprint(f\"Prompt A — proportionality: {results_a['avg_proportionality']:.1f}/10\")\nprint(f\"Prompt B — proportionality: {results_b['avg_proportionality']:.1f}/10\")\n```\n\nPrompt B scored higher on Nether scenarios because it explicitly considers biome danger. That insight came from the eval — not from vibes.\n\n```\nObserve (interactions.json)\n    ↓\nEvaluate (code checks + LLM judge)\n    ↓\nAdjust (modify system prompt)\n    ↓\nMeasure (re-run eval, compare scores)\n    ↓\nDeploy (update prompt in mock server)\n    ↓\nObserve again...\n```\n\nThis is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod. The pattern is universal.\n\nAfter running evals across 20 scenarios:\n\n| Metric | Prompt v1 | Prompt v2 (biome-aware) |\n|---|---|---|\n| Code pass rate | 95% | 95% |\n| Threat assessment | 7.2/10 | 8.4/10 |\n| Proportionality | 6.8/10 | 8.1/10 |\n| Reason quality | 7.5/10 | 7.8/10 |\n\nThe biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.\n\nThe 5% code failure rate? JSON parsing issues when the model occasionally adds explanation text before the JSON. Fixed by making the parser more robust (find first `{`\n\n, last `}`\n\n).\n\n| Eval Concept | What It Teaches |\n|---|---|\n| Code evaluators | Input validation, schema enforcement |\n| LLM-as-Judge | AI evaluating AI (meta-evaluation) |\n| Eval dataset | Test cases for non-deterministic systems |\n| A/B testing prompts | Experimentation, data-driven decisions |\n| Feedback loop | Continuous improvement, observability |\n| Scoring thresholds | Quality gates, SLOs for AI |\n| interactions.json | Observability, audit trails |\n\nAcross 10 blog posts, this mod went from \"7 swords with cloud names\" to a full system with:\n\nAll teaching cloud computing through gameplay. All open source.\n\n*This concludes the Cloud Swords blog series. 10 posts, one mod, and more cloud concepts than most certification courses. Thanks for reading.*\n\n**Code:**\n\n*I'm Carlos Cortez — and this is Breaking the Cloud.*", "url": "https://wpnews.pro/news/blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores", "canonical_source": "https://dev.to/ccortezb/-blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores-to-make-it-a-333j", "published_at": "2026-08-01 16:13:45+00:00", "updated_at": "2026-08-01 16:44:20.007406+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "ai-tools"], "entities": ["Minecraft", "Strands Agent", "gpt-oss:20b"], "alternates": {"html": "https://wpnews.pro/news/blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores", "markdown": "https://wpnews.pro/news/blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores.md", "text": "https://wpnews.pro/news/blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores.txt", "jsonld": "https://wpnews.pro/news/blog-10-minecraft-mod-agent-i-made-my-ai-agent-judge-itself-and-used-the-scores.jsonld"}}