Blog 10: Minecraft Mod Agent- I Made My AI Agent Judge Itself — And Used the Scores to Make It a Better Combat Advisor Cloud Swords Mod — Blog 10 In Blog 9, I gave my Minecraft mod an AI brain.
A Strands Agent that decides how many allies to summon based on your combat situation.
It works.
But "works" isn't a metric.
How do I know the AI is making good decisions?
When a player has 3 hearts and 8 zombies closing in, does the agent summon 5 allies (correct) or 1 (death sentence)?
When the player is full health with no threats, does it correctly do nothing — or waste resources summoning minions for no reason?
I needed an evaluation framework.
So I built one.
And the judge?
Another AI.
The Problem: Non-Deterministic Decisions The agent uses .
Same input can produce slightly different outputs between calls.
You can't write because there's no single correct answer.
What you CAN evaluate: Is the action valid JSON? (deterministic check) Is the action within bounds? (1-5 minions, valid buff types) Is the response proportional to the threat? (requires judgment) Is the reasoning coherent? (requires judgment) The first two are code.
The last two need an LLM.
The Eval Dataset: Real Interactions Remember from the mock server?
Every time the agent makes a decision, it's logged: I took 20+ real interactions and added expected behavior ranges: Level 1: Code Evaluators (Free, Instant) These catch 60% of problems instantly: malformed JSON, out-of-bounds values, missing reasons, wildly disproportionate responses.
Level 2: LLM-as-Judge For the subjective stuff — "is this a good decision?" — I use the same model as a judge: The meta part: the same AI model that makes the decisions is also judging them.
This works because the judge has the expected behavior as reference — it's not evaluating in a vacuum.
The Eval Runner A/B Testing System Prompts The real power: I can test different system prompts and measure which one produces better decisions.
Prompt B scored higher on Nether scenarios because it explicitly considers biome danger.
That insight came from the eval — not from vibes.
The Feedback Loop This is the same loop from the LLMOps series (Part 3: Eval-Driven Development) — but applied to a Minecraft mod.
The pattern is universal.
Results: What I Found After running evals across 20 scenarios: Metric Prompt v1 Prompt v2 (biome-aware) Code pass rate 95% 95% Threat assessment 7.2/10 8.4/10 Proportionality 6.8/10 8.1/10 Reason quality 7.5/10 7.8/10 The biome-aware prompt improved threat assessment by 1.2 points — specifically in Nether and End scenarios where v1 was under-responding.
The 5% code failure rate?
JSON parsing issues when the model occasionally adds explanation text before the JSON.
Fixed by making the parser more robust (find first , last ).
Cloud/AI Concepts Taught Eval Concept What It Teaches Code evaluators Input validation, schema enforcement LLM-as-Judge AI evaluating AI (meta-evaluation) Eval dataset Test cases for non-deterministic systems A/B testing prompts Experimentation, data-driven decisions Feedback loop Continuous improvement, observability Scoring thresholds Quality gates, SLOs for AI interactions.json Observability, audit trails What I'd Do Next Automated eval in CI — every prompt change triggers eval, blocks if scores drop Player feedback — thumbs up/down after each cloud invocation, feeds back into dataset Seasonal prompts — different system prompts for different game phases (early/mid/endgame) Multi-model comparison — test gpt-oss:20b vs Claude Haiku vs Nova Lite for cost/quality tradeoff The Full Picture Across 10 blog posts, this mod went from "7 swords with cloud names" to a full system with: Custom textures generated with Python + AI Energy systems with BFS algorithms Multiblock machines with custom GUIs A spell system with 28 spells Armor sets with dodge mechanics Wandering merchants with elite progression A serverless AI backend An evaluation framework for AI decisions All teaching cloud computing through game