Part 1 of "Multi-Agent Systems in Production: What They Don't Tell You" — a four-part series following the saga of Horcrux Hunt, a multi-agent Harry Potter game that taught me everything about production AI the expensive way.
I built Horcrux Hunt, an interactive Harry Potter-themed game where two AI agents battle each other live in front of an audience. Harry (the protagonist, powered by Claude on Amazon Bedrock via Strands SDK) hunts Horcruxes hidden across 15 locations. Voldemort (the adversary) relocates them, plants decoys, and corrupts Harry's beliefs.
Think of it as adversarial hide-and-seek between two LLMs, with a live audience voting and watching Harry's search unfold in real-time on a Streamlit dashboard.
It was supposed to be a fun weekend demo. The audience loved it. The CloudWatch metrics did not.
The bill: $1,847 for one weekend.
More than my rent.
And the performance was terrible:
The audience feedback said things like "game is slow" and "Harry seldom wins." They didn't know it was also bankrupting me.
We build multi-agents systems. We keep adding agents for every task. But we never ask - "How many agents are too many" ? Here's the formula nobody shows you when they pitch multi-agent architectures:
Cost = tokens × agents × turns × retries × context_replay
Each term multiplies the others. It's not additive, it's multiplicative. Let me break down why, using Horcrux Hunt as the anatomy lesson.
Every turn of Horcrux Hunt, the game feeds Harry the entire conversation history like every search, every signal, every ally ability used. By turn 50, that's a LOT:
Every single turn, the LLM re-reads the entire conversation history. It's like Harry reading his complete mission journal from page one every time he makes a decision. By turn 50, you're paying 7.5× what turn 1 cost and that's just one agent.
With two agents (Harry AND Voldemort), you have 2× the token curve. Each agent maintaining its own inflating context.
Most people focus on input tokens. But output tokens cost 5× more than input tokens on most models (Claude 3 Sonnet: $0.003/1K input vs $0.015/1K output). And they're sequential costing 12ms per token, and can't be parallelized.
Harry's responses averaged 150-200 output tokens per turn. Voldemort's averaged 100-150. Across 50 turns × 2 agents, output tokens accounted for 39% of wall-clock time. The audience was waiting for token generation, not thinking.
What I imagined a turn looked like:
What a turn actually required:
50 turns × 12 operations = 600 operations per game. The sequential nature is the real bottleneck. You can't parallelize Harry and Voldemort because Voldemort's action depends on what Harry just did.
Here's what I noticed when I analyzed my retry rate: 15% of LLM responses failed validation. Harry would produce an invalid action format, or try to search a location on cooldown, or attempt to use an ally ability he'd already spent.
Each failure: 3 seconds of reasoning → rejected → full context replayed → try again. Each retry cost full token price.
The system was discovering failures after spending money. Failing slow (3 seconds into inference). Failing expensive (full token cost per failure). I'd later name the opposite of this pattern.
For a game. A two-agent game where Harry chases Horcruxes.
The cost breakdown:
Now imagine deploying this as an always-on interactive experience. Imagine scaling to 1,000 simultaneous games. The numbers get terrifying fast.
I didn't need fewer agents. I needed fewer expensive decisions. Here's the optimization ladder pushing decisions DOWN from the costly LLM layer to cheaper alternatives.
Before: Harry saw 90 possible actions per turn (15 locations × 6 action types: search, attack, use_ally, investigate, fortify, retreat) and the LLM had to reason about which were valid.
After: A constraint solver prunes invalid actions before Harry's LLM sees them:
def get_valid_actions(game_state, agent="harry"):
actions = []
for loc in game_state.locations:
if game_state.cooldown[loc] == 0: # not on cooldown
if game_state.usage_count[loc] < MAX_USES: # not exhausted
if game_state.budget > 0: # has action budget
actions.append(loc)
return actions # typically 2-4 options, not 90
Result: 90 options → 2-4 valid options. Harry's LLM never wastes tokens reasoning about locations on cooldown or abilities already spent. 80% reduction in invalid reasoning tokens.
This is where I coined the principle: "Fail Fast, Fail Free."
The constraint solver catches bad decisions before they touch the LLM. An invalid action rejected by a 0.2ms Python function costs nothing. The same invalid action reasoned about by Claude for 3 seconds costs tokens, latency, and often a retry when post-inference validation fails.
Fail fast = catch it early. Fail free = catch it before the meter starts running.
Before: Harry re-read 50 turns of narrative history (5,000 tokens) to figure out "where is the Horcrux probably located?"
Turn 1: Harry searched Hogwarts → negative signal
Turn 2: Harry searched Diagon Alley → positive signal
Turn 3: Harry attacked Diagon Alley → decoy!
Turn 4: Voldemort relocated something...
... (50 turns of this = 5,000 tokens)
After: A Bayesian belief map computes probabilities outside the LLM:
class HorcruxBeliefMap:
def __init__(self, locations):
n = len(locations)
self.beliefs = {loc: 1.0/n for loc in locations} # uniform prior
def update_on_signal(self, loc, signal):
if signal == "positive":
self.beliefs[loc] *= 3.0 # 3x more likely here
elif signal == "negative":
self.beliefs[loc] *= 0.1 # 10x less likely
elif signal == "destroyed":
self.beliefs[loc] = 0.0 # confirmed eliminated
self.normalize()
def top_targets(self, n=3):
sorted_locs = sorted(self.beliefs.items(), key=lambda x: -x[1])
return sorted_locs[:n]
What Harry's LLM actually sees: "top_target: Hogwarts (p=0.34), Azkaban (p=0.22)"
= 15 tokens, not 5,000.
Result: 97% context reduction. Cost drops from $0.015 to effectively $0 for the belief computation. And Harry makes better decisions because probabilities are more precise than narrative intuition.
Another face of "Fail Fast, Fail Free" is if you can compute the answer with math ($0), why would you pay an LLM ($0.015) to infer it from narrative text?
Not every Voldemort decision needs a $200B parameter model. Some game situations have obvious optimal moves:
def voldemort_decide(game_state):
if game_state.harry_adjacent_to_horcrux():
return RelocateAction(game_state.threatened_horcrux)
if game_state.decoy_budget > 0 and harry_entropy(game_state) < 1.5:
return PlantDecoyAction(game_state.harry_top_target)
if game_state.turn < 10 and harry_entropy(game_state) > 3.0:
return WaitAction()
return call_llm_for_strategy(game_state)
Result: 60% of Voldemort's decisions handled with zero LLM cost. Only the genuinely strategic moments where multiple valid strategies compete, justify an inference call.
The most impactful change was structural. I redesigned Horcrux Hunt so only 2 of 8 modules ever touch the LLM:
FREE MODULES (6):
├── Game Engine (rules, turn management)
├── Constraint Solver (valid action computation)
├── Belief Manager (Bayesian probability updates)
├── State Persistence (DynamoDB read/write)
├── Validation Layer (response format checking)
└── Metrics & Logging (CloudWatch, dashboards)
EXPENSIVE MODULES (2):
├── Harry Strategic Layer (genuinely uncertain decisions)
└── Voldemort Strategic Layer (only when heuristics can't decide)
The Interface Boundary compresses context at the border between free and expensive:
This is "Fail Fast, Fail Free" as architecture: clear cost boundaries. Validation happens in the free zone. If something is going to fail, it fails in one of the 6 free modules, but never in the 2 expensive ones.
Same game. Same Harry. Same Voldemort. Same Claude Sonnet model. Radically different architecture:
| Metric | Before | After | Change |
|---|---|---|---|
| Cost per game | $1.95 | $0.35 | -82% |
| Latency per turn | 12s | 3s | -75% |
| Harry win rate | 23% | 52% | +29pp |
| Timeout rate | 18% | <3% | -83% |
| Retry rate | 15% | <3% | -80% |
| Annual cost at scale | $588K | $102K | -$486K |
Harry wins more, not because the model is smarter, but because he's reasoning over focused, relevant information instead of drowning in 6,000 tokens of noise. The audience sees 3-second turns instead of 12-second waits. The game is actually fun to watch now.
The answer to "How many agents are too many?" isn't a number. It's a question:
"Is this decision worth an LLM call?"
Most of Voldemort's decisions weren't. Most of Harry's probability calculations weren't. Most of the validation logic wasn't. Once I stopped paying the LLM to discover things I already knew, the costs fell off a cliff.
The Optimization Ladder:
Push every decision as far DOWN that ladder as it can go. Fail fast, fail free at every layer.
🚀 What's Next
The bill is under control. But Harry's still losing.
He searches the same location twice. He forgets signals from 3 turns ago. He contradicts his own belief map. The problem isn't cost anymore, it's memory masquerading as reasoning failure.
→ Part 0: Fail Fast, Fail Free
Read this blog to know more about the principle.
→ Part 2: Why Your Agent Forgets where I discover that 68% of Harry's "reasoning failures" are actually retrieval failures, and the fix is entropy math, not bigger models.
Spoiler: I compressed Harry's context from 12,000 tokens to 340. Same LLM. Same prompts. Completely different agent.
💬 Quick diagnostic for your agent:
Run the same input 5 times. Does your agent give the same output every time?
✅ Yes → You probably have a cost or integration problem (Part 1 or 3)
❌ No → You definitely have a memory problem (next post)
🤷 Haven't checked → ...that's the scariest answer of all
Drop your answer below. I'll tell you exactly which post in this series has your fix. 👇
🔖 Follow me to get Part 2 when it drops or keep refreshing your inference bill until you're motivated enough to read it. Your choice. 💸
I am a Gen AI Developer Advocate & Architect. I built a multi-agent AI game to entertain a conference audience and accidentally created the most expensive stress test for multi-agent systems I'd ever seen. So. I adapted the classic "Fail Safe" and came up with "Fail Fast, Fail Free" after that $1,847 weekend taught me that the most expensive failure is one you discover too late.