Benchmarking LLMs' Swarm Intelligence

Researchers introduced SwarmBench, a novel benchmark for evaluating the swarm intelligence of large language models (LLMs) acting as decentralized agents, testing five foundational multi-agent tasks: Pursuit, Synchronization, Foraging, Flocking, and Transport. The benchmark uses a configurable 2D grid environment with limited local perception and communication, and includes an open-source toolkit with environments, prompts, and evaluation scripts. Initial results show that models like GPT-4o-mini and Llama-3.1-70B exhibit emergent coordination, but performance varies by task, highlighting current limitations in LLM-based swarm coordination.

Figure 1: Natural Swarm Intelligence Inspiration and SwarmBench Tasks. SwarmBench is a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of Large Language Models LLMs acting as decentralized agents. It addresses the gap in existing benchmarks by focusing on scenarios with strict constraints, such as limited local perception and communication, characteristic of natural swarms. Large Language Models LLMs show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems MAS under such constraints remains largely unexplored. SwarmBench aims to bridge this gap. Figure 2: Conceptual Architecture of SwarmBench. Figure 3: Overview of LLM Performance on SwarmBench Tasks - Novel Benchmark: Grounded in swarm intelligence principles, assessing emergent decentralized coordination. - Five Foundational MAS Tasks: - 🎯 Pursuit: Agents collaboratively track and corner a faster-moving prey. - ⏱️ Synchronization: Agents aim to synchronize an internal binary state across the swarm. - 🧺 Foraging: Agents navigate to find a food source, transport it to a nest, and coordinate task allocation. - 🐦 Flocking: Agents must move as a cohesive group, maintaining alignment and separation. - 🚚 Transport: Multiple agents must cooperate to push a large object towards a designated goal. - 🎯 - Configurable 2D Grid Environment: Forces agents to rely on local sensory input k × k view and local communication. - Systematic Evaluation: Characterizes current LLM abilities and limitations in canonical swarm scenarios. - Analysis of Emergent Group Dynamics: Connects LLM swarm behavior to established collective intelligence concepts. - Open-Source Toolkit: - Customizable and scalable physical system. - Environments, standardized prompts, and evaluation scripts. - Comprehensive experimental datasets. - Designed for reproducible research into LLM-based MAS coordination. Ensure you have Conda installed. Then, create the environment using the provided file: conda env create -f environment.yaml conda activate swarmbench Execute the eval.py script to run the benchmark evaluations: python eval.py python from swarmbench import SwarmFramework if name == ' main ': name = 1 for task in 'Transport', 'Pursuit' : {'Transport', 'Pursuit', 'Synchronization', 'Foraging', 'Flocking'} for model in 'gpt-4o-mini', 'llama-3.1-70b' : Models for seed in 27, 42 : SwarmFramework.submit f'exp {name}', SwarmFramework.model config model, 'YOUR API KEY', 'YOUR API BASE' , API task, log dir='YOUR LOG DIR', Logging num agents=10, max round=100, width=10, height=10, seed=seed, view size=5 name += 1 SwarmFramework.run all max parallel=4 This will run the LLM agents through the defined tasks and log the results. python load dataset.py After running evaluations, you can generate replay videos of the simulations: ./export videos.sh This script will process the log files from your experiments by default in ./experiment outputs/ and generate MP4 videos. Example Replay: o4-mini, the best out of 5 runs. o4-mini, the best out of 5 runs. o4-mini, the best out of 5 runs. o4-mini, the best out of 5 runs. o4-mini, the best out of 5 runs. For a more detailed, step-by-step analysis, you can use the interactive replay tool: python analysis/replay interactive latex.py -v --log-dir your experiment dir Replace your experiment dir with the actual directory of the experiment you want to replay. Screenshot of the interactive replay interface. python analysis/score agg.py --log-dir your experiment dir You will get output like this: Model | Avg Score | ± Std Dev | Games ----------------------------------------------------------- o4-mini | 9.60 | ± 0.49 | 5 gemini-2.0-flash | 8.80 | ± 1.60 | 5 gpt-4.1 | 8.40 | ± 1.85 | 5 claude-3-7-sonnet-20250219 | 4.40 | ± 1.20 | 5 DeepSeek-V3 | 4.20 | ± 2.48 | 5 o3-mini | 3.60 | ± 2.06 | 5 gpt-4o | 3.40 | ± 1.50 | 5 qwen/qwq-32b | 2.20 | ± 1.94 | 5 Meta-Llama-3.1-70B-Instruct | 1.80 | ± 0.40 | 5 gpt-4.1-mini | 1.40 | ± 0.80 | 5 meta-llama/llama-4-scout | 1.20 | ± 0.75 | 5 deepseek-ai/DeepSeek-R1 | 1.00 | ± 0.63 | 5 claude-3-5-haiku-20241022 | 0.60 | ± 0.49 | 5 ----------------------------------------------------------- Run this script to export the plots. The output will be saved in ./figs ./export metrics plots.sh For a single task: python analysis/task score trends.py --log-dir experiment v01 --pad-rounds 100 For multiple tasks: python analysis/task score trends.py --log-dir experiment v01 experiment v02 experiment v03 experiment v04 experiment v05 --multi-plot --pad-rounds 100 In this project we showcase a simple implementation for swarm RL training. We made a synchronous version adapted from the SwarmBench, which provides a way to extract prompts from the swarm environment, generate responses and feed them back to the environment, allowing us to train LLM in OpenRLHF https://github.com/OpenRLHF/OpenRLHF or other fine-tuning framework. See 🍯 SwarmRLVR /RUC-GSAI/YuLan-SwarmIntell/blob/main/swarmRLVR Here's a list of some stuff that will be added to the codebase - Include full dataset - Add RLVR training & testing code - RLVR trained small model for demo We enthusiastically welcome community contributions to SwarmBench If you're passionate about advancing research in LLM swarm intelligence, we invite you to help improve the benchmark by proposing new tasks, implementing features, etc. Your insights are highly valued. For significant intellectual contributions that substantially advance the project, we are open to exploring collaborative opportunities on future research and potential academic publications, in line with standard academic practices. Please engage with us via GitHub Issues for discussions and submit Pull Requests for your contributions. @article{ruan2025benchmarkingllmsswarmintelligence, title={Benchmarking LLMs' Swarm intelligence}, author={Kai Ruan and Mowen Huang and Ji-Rong Wen and Hao Sun}, year={2025}, eprint={2505.04364}, archivePrefix={arXiv}, primaryClass={cs.MA}, url={https://arxiv.org/abs/2505.04364}, }