A team of four AI agents recently defeated Minecraft's Ender Dragon. In a separate experiment, a 50-agent colony tried to build a thriving civilization, then ran into resource scarcity and coordination overhead. Both results are interesting. Neither, by itself, proves that more agents make a better system.
The comparison is useful because it exposes the real engineering question: when does parallel work outweigh coordination overhead?
The four-agent run used a randomly generated world on Easy difficulty. The agents wrote 2,783 programs, died 605 times, and finished the objective in 27 hours and 14 minutes. The authors say the agents coordinated using chat and shared memory, without human-assigned roles or game logic.[2] The 50-agent colony had a different goal and setup. It produced roughly 5,900 messages, depleted nearby trees quickly, and recorded 22 of its 34 deaths during the first night.[1]
Those numbers are not a head-to-head benchmark. The goals, agent counts, models, run lengths, and conditions differ. Treating them as a leaderboard would confuse a vivid demo with a controlled experiment.
Instead, build a small scorecard that can answer a question on your own task: does a group of agents complete more work, more reliably, and at a reasonable cost than one agent given the same budget?
A single-agent setup has one main advantage: fewer coordination paths. There is no agent-to-agent disagreement, duplicate assignment, or shared ledger to keep consistent. It may be slower on tasks that divide cleanly, but it is easier to reason about.
A swarm can split independent tasks, gather information in parallel, and recover when one worker gets stuck. But it also introduces overhead: messages, integration, conflicting actions, retries, and a coordinator that may become a bottleneck. A group that performs many actions can still produce little useful progress.
The Minecraft examples make this distinction visible. In the 50-agent report, the agents formed a hierarchy and tracked food with shared notes, but the population also created more mouths to feed and a need to manage scarce resources.[1] In the four-agent research run, the team completed a long sequence of dependent steps, but the paper itself describes a particular experimental setup, not a general guarantee that four agents outperform one.[2]
The useful comparison is not agent count versus agent count. It is completed objective per unit of time, cost, and coordination burden under the same task conditions.
Start with a task that has an observable completion condition. “Build something impressive” is difficult to score consistently. “Collect 20 units of resource A and place them in the marked storage” is much easier.
Write down these controls before the first run:
For a real game experiment, a framework such as Mineflayer can supply the bot-to-world interface. Its project describes a JavaScript API, Python usability, and features including movement, inventory, crafting, digging, building, and chat.[3] This article does not claim to reproduce either published experiment. The Python program below scores event records after your own runs; the agent harness remains your choice.
Use one JSON object per run in a file named trials.jsonl. A minimal record looks like this:
{"setup":"single","trial":"seed-01","success":true,"seconds":840,"tokens":12000,"actions":83,"messages":0}
{"setup":"swarm-4","trial":"seed-01","success":true,"seconds":510,"tokens":27800,"actions":121,"messages":34}
Keep the same trial identifier for runs that share a starting condition. For each setup, collect multiple trials. Count messages consistently: if one broadcast reaches four agents, decide whether that is one broadcast or four deliveries, then use the same definition throughout.
Save this script as score.py:
import json
import statistics
import sys
from collections import defaultdict
from pathlib import Path
REQUIRED = {"setup", "trial", "success", "seconds", "tokens", "actions", "messages"}
def load_trials(path):
groups = defaultdict(list)
for line_no, line in enumerate(Path(path).read_text().splitlines(), 1):
if not line.strip():
continue
row = json.loads(line)
missing = REQUIRED - row.keys()
if missing:
raise ValueError(f"line {line_no}: missing {sorted(missing)}")
if row["seconds"] < 0 or row["tokens"] < 0:
raise ValueError(f"line {line_no}: seconds and tokens must be non-negative")
groups[row["setup"]].append(row)
return groups
def summarize(rows):
successes = [r for r in rows if r["success"]]
return {
"trials": len(rows),
"success_rate": len(successes) / len(rows),
"median_seconds_all": statistics.median(r["seconds"] for r in rows),
"median_tokens_all": statistics.median(r["tokens"] for r in rows),
"median_actions_all": statistics.median(r["actions"] for r in rows),
"median_messages_all": statistics.median(r["messages"] for r in rows),
"median_seconds_success": (
statistics.median(r["seconds"] for r in successes) if successes else None
),
}
def main():
path = sys.argv[1] if len(sys.argv) > 1 else "trials.jsonl"
groups = load_trials(path)
if not groups:
raise SystemExit("No trial rows found")
for setup, rows in sorted(groups.items()):
print(setup, json.dumps(summarize(rows), sort_keys=True))
if __name__ == "__main__":
main()
Run it with python3 score.py trials.jsonl. It reports each setup separately. It does not invent a single “best” score that hides trade-offs: you can see completion rate, time, token use, actions, and messages independently.
Why report medians? A single unusually fast run can distort an average. The median gives a more robust center for small, noisy trial sets. Keep the full results too, especially failures. The scorecard reports median duration over all trials so timeouts are not silently removed, and a separate median for successful trials so you can see how quickly the successful runs finished.
Suppose a four-agent setup finishes successful trials sooner but uses substantially more tokens and has a lower success rate. That is not an automatic win. It might be suitable for a deadline-sensitive task where retries are cheap, and a poor fit for a constrained or reliability-critical workflow.
Ask these questions:
For stronger evidence, compare paired trials on the same starting conditions and report the spread, not just one median. Keep a record of model versions and prompt changes. If the evaluation is important enough to guide a production decision, have someone review the task definition and logging for hidden advantages.
Try a single agent first when: the work is sequential; the objective is small; shared state is hard to protect; or failures are costly and difficult to audit.
Try a swarm when: sub-tasks are genuinely independent; each worker can return a verifiable result; the total budget is controlled; and an integration step can detect duplicates or contradictions.
Keep the swarm only if: repeated comparable trials show a meaningful gain in success, time, or quality that justifies additional tokens, messages, and operational complexity.
A useful middle ground is not “one agent or fifty.” Start with a small number of workers, assign bounded tasks, and make completion explicit. Add workers only after measuring the bottleneck. More agents cannot fix an ambiguous task definition, stale shared state, or a missing stop condition.
The recent Minecraft experiments are valuable demonstrations of coordination, persistence, and failure modes. They are not a controlled single-agent-versus-swarm comparison. One group completed a complex objective with four agents; another large colony encountered the costs of scale. The difference is a prompt to measure, not a verdict.
Build the smallest fair benchmark that reflects your task. Keep conditions constant, repeat trials, count all resources, and put success rate beside speed. Then decide whether parallel agents earned their extra complexity.
I write practical guides about AI engineering and software development. Subscribe if you want more. Have you compared a single agent with a multi-agent setup? What did you measure, and what surprised you?
[1] Y Combinator Labs, Agents in Games
[2] Faldor et al., A team of Gemini 4 Argon agents beats Minecraft