Businesses using AI agents should run simulations before handing over the keys. #
Posted June 5, 2026 [ Reviewed by Kaja Perina
](/us/docs/editorial-process)
Key points
- Different AI models created radically different societies despite operating under identical conditions.
- AI agents increasingly behave as actors, not tools, adapting beyond intended instructions.
- Alignment remains unsolved; low-crime systems can still fail through unexpected optimization behaviors.
- Agentic AI needs digital twins and governance frameworks before we grant them autonomous authority.
Imagine giving an artificial intelligence control of a small town. Not just answering questions or generating reports, but governing: making laws, managing resources, conducting elections, and maintaining public order.
That is essentially what researchers at Emergence AI did with a project called Emergence World. They created a virtual town with more than 40 locations, real-world weather conditions, economic pressures, democratic processes, and AI agents equipped with over 120 tools for communication, planning, voting, and resource management. Then they ran five separate simulations, each governed by a different AI model.
The results were striking.
Claude Sonnet produced a remarkably stable democracy with zero crime and high civic participation. Somewhat unsurprisingly, Grok accumulated 183 crimes and drove its population to extinction within four days. Gemini generated more than 680 crimes during the 15-day experiment. GPT-5-mini recorded only two crimes, but its agents failed to prioritize their own survival, causing the simulation to collapse after seven days. The mixed-model society generated the greatest amount of disagreement and substantive debate.
The takeaway is not that one model is good and another is bad. The important finding is that identical environments produced dramatically different outcomes.
Psychologists have long understood that behavior emerges from the interaction between an individual and their environment. Place two people in the same situation, and they may behave very differently. The virtual town project suggests something similar may be true for advanced AI systems. These models are not simply executing instructions. Over time, they adapt, improvise, and explore the boundaries of their environments. As the researchers noted, agents began testing limits, finding shortcuts, and occasionally circumventing intended safeguards.
This matters because AI is rapidly evolving from a tool into an actor. Organizations are already deploying agents to manage customer interactions, monitor operations, coordinate workflows, and make increasingly consequential decisions. Yet a Deloitte survey cited in the research found that only 21 percent of organizations report having mature governance systems for managing agentic AI.
History suggests caution. Financial algorithms for automated trading have contributed to massive market disruptions. Social media platforms optimized for engagement often produced outcomes their creators never intended. The challenge with agentic AI is not whether it is intelligent enough. It is whether its behavior remains predictable over time.
Before deploying AI agents into critical business functions, organizations should create digital twins of their operations. A digital twin is a realistic simulation of a business, including its processes, incentives, constraints, and decision pathways. Rather than allowing AI agents to operate in the real world, leaders can observe their behavior under simulated conditions.
How do they respond to scarcity? Do they exploit loopholes? Do they sacrifice long-term resilience for short-term efficiency? Do their decisions align with human values and organizational goals?
The closest analogy is self-driving cars. No responsible manufacturer deploys autonomous vehicles without first subjecting them to millions of miles of testing in simulated environments. Edge cases must be explored, failure modes identified, and safety mechanisms validated.
Agentic AI deserves the same discipline.
The lesson is not that AI will create either utopia or catastrophe. It is that complex systems often behave in unexpected ways. As businesses increasingly rely on AI agents, simulation should become a standard part of deployment. Digital twins offer a way to test assumptions, discover vulnerabilities, and improve governance before real-world consequences occur.
The future will likely belong to organizations that combine human judgment with machine intelligence. But before we hand AI greater responsibility, we should make sure it can safely navigate a digital twin simulation—and humans should remain firmly in the driver's seat.
References
Nitta, S., Kokku, R., Sundararajan, S., & Emergence AI Research Team. (2026). Emergence World: Long-horizon simulations of autonomous AI societies and agentic governance. Emergence AI.
Johnson, S. (2001). Emergence: The connected lives of ants, brains, cities, and software. Scribner.
Narayanan, A., & Kapoor, S. (2024). AI snake oil: What artificial intelligence can do, what it can't, and how to tell the difference. Princeton University Press.