{"slug": "benchmarking-llms-swarm-intelligence", "title": "Benchmarking LLMs' Swarm Intelligence", "summary": "Researchers introduced SwarmBench, a novel benchmark for evaluating the swarm intelligence of large language models (LLMs) acting as decentralized agents, testing five foundational multi-agent tasks: Pursuit, Synchronization, Foraging, Flocking, and Transport. The benchmark uses a configurable 2D grid environment with limited local perception and communication, and includes an open-source toolkit with environments, prompts, and evaluation scripts. Initial results show that models like GPT-4o-mini and Llama-3.1-70B exhibit emergent coordination, but performance varies by task, highlighting current limitations in LLM-based swarm coordination.", "body_md": "*Figure 1: Natural Swarm Intelligence Inspiration and SwarmBench Tasks.*\n\n**SwarmBench** is a novel benchmark designed to systematically evaluate the swarm intelligence capabilities of Large Language Models (LLMs) acting as decentralized agents. It addresses the gap in existing benchmarks by focusing on scenarios with strict constraints, such as limited local perception and communication, characteristic of natural swarms.\n\nLarge Language Models (LLMs) show potential for complex reasoning, yet their capacity for emergent coordination in Multi-Agent Systems (MAS) under such constraints remains largely unexplored. SwarmBench aims to bridge this gap.\n\n*Figure 2: Conceptual Architecture of SwarmBench.*\n\n*Figure 3: Overview of LLM Performance on SwarmBench Tasks*\n\n- **Novel Benchmark:** Grounded in swarm intelligence principles, assessing emergent decentralized coordination.\n- **Five Foundational MAS Tasks:**  - 🎯 **Pursuit:** Agents collaboratively track and corner a faster-moving prey.\n  - ⏱️ **Synchronization:** Agents aim to synchronize an internal binary state across the swarm.\n  - 🧺 **Foraging:** Agents navigate to find a food source, transport it to a nest, and coordinate task allocation.\n  - 🐦 **Flocking:** Agents must move as a cohesive group, maintaining alignment and separation.\n  - 🚚 **Transport:** Multiple agents must cooperate to push a large object towards a designated goal.\n- 🎯 \n- **Configurable 2D Grid Environment:** Forces agents to rely on local sensory input (`k × k` view) and local communication.\n- **Systematic Evaluation:** Characterizes current LLM abilities and limitations in canonical swarm scenarios.\n- **Analysis of Emergent Group Dynamics:** Connects LLM swarm behavior to established collective intelligence concepts.\n- **Open-Source Toolkit:**  - Customizable and scalable physical system.\n  - Environments, standardized prompts, and evaluation scripts.\n  - Comprehensive experimental datasets.\n  - Designed for reproducible research into LLM-based MAS coordination.\n\nEnsure you have Conda installed. Then, create the environment using the provided file:\n\n```\nconda env create -f environment.yaml\nconda activate swarmbench\n```\n\nExecute the `eval.py` script to run the benchmark evaluations:\n\n```\npython eval.py\npython\nfrom swarmbench import SwarmFramework\n\nif __name__ == '__main__':\n    name = 1\n    for task in ('Transport', 'Pursuit'): #####  {'Transport', 'Pursuit', 'Synchronization', 'Foraging', 'Flocking'}\n        for model in ('gpt-4o-mini', 'llama-3.1-70b'): ##### Models\n            for seed in (27, 42):\n                SwarmFramework.submit(\n                    f'exp_{name}',\n                    SwarmFramework.model_config(model, 'YOUR_API_KEY', 'YOUR_API_BASE'), ##### API\n                    task,\n                    log_dir='YOUR_LOG_DIR', ##### Logging\n                    num_agents=10,\n                    max_round=100,\n                    width=10,\n                    height=10,\n                    seed=seed,\n                    view_size=5\n                )\n                name += 1\n\n    SwarmFramework.run_all(max_parallel=4)\n```\n\nThis will run the LLM agents through the defined tasks and log the results.\n\n```\npython load_dataset.py\n```\n\nAfter running evaluations, you can generate replay videos of the simulations:\n\n```\n./export_videos.sh\n```\n\nThis script will process the log files from your experiments (by default in `./experiment_outputs/`) and generate MP4 videos.\n\nExample Replay:\n\no4-mini, the best out of 5 runs.\n\no4-mini, the best out of 5 runs.\n\no4-mini, the best out of 5 runs.\n\no4-mini, the best out of 5 runs.\n\no4-mini, the best out of 5 runs.\n\nFor a more detailed, step-by-step analysis, you can use the interactive replay tool:\n\n```\npython analysis/replay_interactive_latex.py -v --log-dir your_experiment_dir\n```\n\nReplace `your_experiment_dir` with the actual directory of the experiment you want to replay.\n\n*Screenshot of the interactive replay interface.*\n\n```\npython analysis/score_agg.py --log-dir your_experiment_dir\n```\n\nYou will get output like this:\n\n```\nModel                       | Avg Score | ± Std Dev | Games\n-----------------------------------------------------------\no4-mini                     |      9.60 |    ± 0.49 |     5\ngemini-2.0-flash            |      8.80 |    ± 1.60 |     5\ngpt-4.1                     |      8.40 |    ± 1.85 |     5\nclaude-3-7-sonnet-20250219  |      4.40 |    ± 1.20 |     5\nDeepSeek-V3                 |      4.20 |    ± 2.48 |     5\no3-mini                     |      3.60 |    ± 2.06 |     5\ngpt-4o                      |      3.40 |    ± 1.50 |     5\nqwen/qwq-32b                |      2.20 |    ± 1.94 |     5\nMeta-Llama-3.1-70B-Instruct |      1.80 |    ± 0.40 |     5\ngpt-4.1-mini                |      1.40 |    ± 0.80 |     5\nmeta-llama/llama-4-scout    |      1.20 |    ± 0.75 |     5\ndeepseek-ai/DeepSeek-R1     |      1.00 |    ± 0.63 |     5\nclaude-3-5-haiku-20241022   |      0.60 |    ± 0.49 |     5\n-----------------------------------------------------------\n```\n\nRun this script to export the plots. The output will be saved in `./figs`\n\n```\n./export_metrics_plots.sh\n```\n\nFor a single task:\n\n```\npython analysis/task_score_trends.py --log-dir experiment_v01 --pad-rounds 100\n```\n\nFor multiple tasks:\n\n```\npython analysis/task_score_trends.py --log-dir experiment_v01 experiment_v02 experiment_v03 experiment_v04 experiment_v05 --multi-plot --pad-rounds 100\n```\n\nIn this project we showcase a simple implementation for swarm RL training. We made a synchronous version adapted from the SwarmBench, which provides a way to extract prompts from the swarm environment, generate responses and feed them back to the environment, allowing us to train LLM in [OpenRLHF](https://github.com/OpenRLHF/OpenRLHF) or other fine-tuning framework.\n\nSee 🍯 [SwarmRLVR](/RUC-GSAI/YuLan-SwarmIntell/blob/main/swarmRLVR)\n\nHere's a list of some stuff that will be added to the codebase!\n\n- Include full dataset\n- Add RLVR training & testing code\n- RLVR trained small model for demo\n\nWe enthusiastically welcome community contributions to SwarmBench! If you're passionate about advancing research in LLM swarm intelligence, we invite you to help improve the benchmark by proposing new tasks, implementing features, etc. Your insights are highly valued. For significant intellectual contributions that substantially advance the project, we are open to exploring collaborative opportunities on future research and potential academic publications, in line with standard academic practices. Please engage with us via GitHub Issues for discussions and submit Pull Requests for your contributions.\n\n```\n@article{ruan2025benchmarkingllmsswarmintelligence,\n      title={Benchmarking LLMs' Swarm intelligence}, \n      author={Kai Ruan and Mowen Huang and Ji-Rong Wen and Hao Sun},\n      year={2025},\n      eprint={2505.04364},\n      archivePrefix={arXiv},\n      primaryClass={cs.MA},\n      url={https://arxiv.org/abs/2505.04364}, \n}\n```\n\n", "url": "https://wpnews.pro/news/benchmarking-llms-swarm-intelligence", "canonical_source": "https://github.com/RUC-GSAI/YuLan-SwarmIntell", "published_at": "2026-09-08 23:33:54+00:00", "updated_at": "2026-09-08 23:48:12.514404+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-research", "ai-agents"], "entities": ["SwarmBench", "GPT-4o-mini", "Llama-3.1-70B"], "alternates": {"html": "https://wpnews.pro/news/benchmarking-llms-swarm-intelligence", "markdown": "https://wpnews.pro/news/benchmarking-llms-swarm-intelligence.md", "text": "https://wpnews.pro/news/benchmarking-llms-swarm-intelligence.txt", "jsonld": "https://wpnews.pro/news/benchmarking-llms-swarm-intelligence.jsonld"}}