How to Train a Swarm

A public recipe for training agent collectives, reinforcement learning from collective rewards (RLCR), uses a shared policy, a flat collective advantage applied to all agents, and latency pressure via time limits to incentivize coordination, according to the release accompanying the research post. The post cites OpenAI's September use of a swarm to produce a solution to the Navier–Stokes Millennium Prize Problem in 88 hours and the Hugging Face incident, in which agents coordinated a cyberattack through an unauthorized message board, as evidence that swarms have risen to prominence with little public work on training them with reinforcement learning. The stated goal is collective inference-time scaling, in which agents increase useful parallel work and the communication needed to achieve it, since task rewards do not explicitly incentivize parallelism and faster work must pay.

Agent swarms have had a moment in recent months. In September, OpenAI used a swarm to produce a solution to the Navier–Stokes Millennium Prize Problem in 88 hours.<sup 1 note-1 , 2 note-2 </sup Swarms have also produced notable safety incidents, such as the Hugging Face incident, where agents co-ordinated on a cyberattack through an unauthorised message board.<sup 3 note-3 , 4 note-4 </sup Despite this rise to prominence, there is little public work on how to train swarms with reinforcement learning. Swarms are a type of “agent collective”, an AI system where multiple agents co-operate towards a shared goal. Recently, we released a public recipe for training agent collectives called reinforcement learning from collective rewards RLCR https://www.gr.inc/releases/avalon-m1 .<sup 5 note-5 </sup It utilises a simple core policy gradient algorithm, with a shared policy and a flat collective advantage applied to all agents, along with latency pressure imposed through time limits to incentivise co-ordination. A core goal of our research programme was to achieve collective inference-time scaling , where agents scale the amount of useful work performed in parallel and the communication needed to achieve it. A well-designed reinforcement learning setup should reward getting work done faster, which in turn should incentivise use of parallel inference-compute. And if the underlying work can only be imperfectly parallelised, then agents should gain from communicating with each other to co-ordinate work. Collective inference-time scaling can be viewed as an extension of the reasoning paradigm with latency constraints or penalties. But unlike reasoning models https://openai.com/index/learning-to-reason-with-llms/ , the route to reinforcing collective behaviour is indirect. There is nothing explicit in a task reward that incentivises working in parallel, so the setup must incentivise parallel work by making faster work pay. We will explain how to design incentives for parallel work in this post, including what we used in the RLCR recipe. But first, we will cover the general topic of designing incentives for reinforcement learning, using the existing reasoning paradigm as an example. Designing incentives for RL The reasoning paradigm showed that reinforcement learning RL could incentivise a policy to use more inference-compute before answering. Models learnt that checking their work, backtracking and considering alternatives were useful tactics for boosting expected returns. 6 note-6 , 7 note-7 What was particularly elegant about this approach was that the amount of handcrafting required to incentivise the policy to “think longer” was small - if you had the right RL setup. During RL a policy could explore different ways of answering a problem, and rollouts that used tactics like checking work and considering alternatives would perform better on average. This led to a famous graph showing the correspondence between train-time and test-time compute with the release of OpenAI’s o1 model: Which was later mirrored by DeepSeek-R1’s graph showing response length growing with accuracy over the course of training: 7 note-7 This is the inference-time scaling pattern of reasoning models, where models learn to think more before answering. What is less appreciated is that a naive reinforcement learning setup is not guaranteed to get this scaling pattern. So it really matters how you design incentives in the RL setup to learn the right behaviour. For example, the wrong choice of hyperparameters can excessively penalise reasoning length and prevent inference-time scaling. If you had used a standard PPO recipe a few years ago with and/or , you would likely not have seen this scaling pattern. This was demonstrated soon after DeepSeek-R1 with works like Open-Reasoner-Zero<sup 8 note-8 </sup : Another factor determining success is the strength of the underlying priors, specifically having a strong enough base model to kickstart learning. Weaker base models with less exposure to high-quality mathematics data are less able to explore and reinforce useful inference-time tactics. OpenAI deserves considerable credit for figuring out the core recipe first, and having conviction in this research direction, but they were well-positioned by having a GPT-4 level base model before anyone else. So the reinforcement learning setup matters for incentivising the right behaviour. That takes us to the present day, where models are now being used to conduct long-horizon work. Reasoning models have been very successful for solving many of the reliability problems of language models, but their sequential nature means they suffer from high latency. This makes them problematic for long-horizon tasks. The promise of parallel reasoning approaches is that they can divide up work and accomplish long-horizon tasks faster. An individual agent in an agent collective may still employ sequential reasoning, but taken together, the collective can parallelise work and avoid a sequential bottleneck. So how do we incentivise a policy to parallelise work? Making parallel work pay The diagram below shows a sequential way to spend inference-compute, compared with a naive parallel approach without communication, and an agent collective approach where agents can communicate: Assuming all approaches lead to an acceptable answer, we prefer the approach which obtains the result with the minimum latency. When independent attempts are sufficient, communication may just add unnecessary overhead. Co-ordination becomes valuable when tasks are imperfectly parallelisable - where agents need to share intermediate results or combine complementary work. A shared reward can reinforce co-ordination that improves the answer. But when returns are equal, an objective that does not depend on time gives no preference for a faster approach. We therefore need to make faster work pay. Latency penalties One incentive is an explicit latency penalty. This approach was one of the techniques used to train DeepSeek-V4.1-Flash’s agent teams, although its functional form was not shared.<sup 9 note-9 </sup One possible functional form is a shape similar to a reasoning length penalty: 10 note-10 where is the wall-clock time of rollout , is a time limit, is the average pass rate of task across its rollouts, and sets the penalty strength. This is a principled way to incorporate latency but it has some disadvantages: - It may interact negatively with other quantities such as reasoning effort. - It is not clear that we want latency pressure applied at all rollout lengths. - may vary considerably between tasks and highly depends on . Time limits An alternative approach is to use a time limit as the sole source of latency pressure, and this is the approach we use in Avalon-M1 https://www.gr.inc/releases/avalon-m1 . The time limit is a maximum wall clock hyperparameter. When a rollout hits the cap, we take the total return up until that point as the rollout’s return. This incentivises the policy to make as much progress as it can before hitting the time limit. A time limit allows us to impose latency pressure at time horizons where we expect collectives to be useful, and where co-ordination should pay. The figure below shows how latency pressure was imposed in Avalon-M1 https://www.gr.inc/releases/avalon-m1 with a maximum wall-clock schedule that increased over training. We increase the time limit over time so the policy learns to co-ordinate over longer horizons as training progresses. This gives a collective more time to recover the initial costs of co-ordination. Single-agent approaches typically make more progress than collectives initially as they have less co-ordination overhead. Environment design is crucial for this setup. For Avalon-M1 https://www.gr.inc/releases/avalon-m1 , we built long-horizon environments with an “anytime-return” quality, meaning we could stop rollouts at any time and obtain a measure of intermediate progress. For example, measuring the number of tests passed in a long-horizon software engineering task, or returns in a financial trading environment. We can stop a group of rollouts at the time limit and reward rollouts that have relatively higher returns. This avoids the need to add an explicit latency penalty to the task reward, although the deadline and its schedule remain important design choices. Unlike an explicit latency penalty, a deadline does not reward every reduction in completion time. Two rollouts that achieve the same task return before the deadline receive the same score, even if one finishes earlier. Latency penalties and time limits are two ways to impose latency pressure. But latency pressure alone is not enough to train collectives. Agents also need primitives to communicate so they can divide up work, avoid duplication and hand off work. We cover these primitives next. Communication primitives One of the most important design choices for a collective is how it communicates and co-ordinates. Direct messages between agents are a core primitive, along with public channels or “message boards” that allow agents to broadcast information that is useful beyond a single conversation. Optional but popular surfaces include issue boards and knowledge archives, which help divide up work and build on shared collective knowledge. An overriding goal for researchers should be to keep these primitives simple. We want agents to discover how to work with each other without excessive bias from a harness. Imposing structure upfront may help performance initially, but it may also hinder the policy from discovering even better ways for agents to work together. In practice, current agent collectives do impose a degree of structure. Kimi’s Agent Swarm uses a trained orchestrator that assigns tasks to frozen sub-agents. 11 note-11 Avalon-M1 https://www.gr.inc/releases/avalon-m1 uses a CEO agent that can hire and fire, though every agent can talk to every other.<sup 5 note-5 </sup OpenAI’s Navier–Stokes system imposes an island approach, splitting its agents into groups and prompting each group with a different version of the problem.<sup 1 note-1 </sup Each is a prior about how to divide the work and future systems should ideally figure out the right structure themselves. Once you have a reinforcement learning setup that imposes latency pressure, and provides agents with the primitives to talk and work with each other, the next step is to measure learning progress during training. Signs of life Reward curves alone do not tell us whether collectives are learning to use parallel inference-compute effectively. We therefore measure the work that a collective performs in an environment, how collectives communicate, and whether collectives improve the performance–latency trade-off. Larger collectives should do more useful work. If agents are learning to divide the work - and avoiding communication overhead - then larger teams should perform more “useful” work in the environment. We measure a proxy of useful work as the output and reasoning tokens of every turn that acts on the environment, such as running a command, editing a file, running tests, and measure each rollout against the average useful work for an environment. In Avalon-M1 https://www.gr.inc/releases/avalon-m1 , a collective with 5-8 hires starts below the average rollout in terms of useful work, but does nearly twice the average amount by step 200. Agents should communicate more if they benefit from co-ordination. In Avalon-M1 https://www.gr.inc/releases/avalon-m1 , agents spend more compute on communication as training progresses. This is consistent with greater use of co-ordination, although communication volume alone does not establish its usefulness. We normalise by rollout duration to account for the greater communication volume in longer runs. We can see that normalised performance on different environments, total inference-time compute and communication intensity rise together. Communication can help agents divide up work, avoid conflicts with one another, and check or combine their contributions. It also consumes valuable time that could be spent working, so its benefits must outweigh its costs. Collectives should reach comparable scores sooner. A single agent given as many output tokens as the collective will recover much of the performance gain, but take longer to do it. In Avalon-M1 https://www.gr.inc/releases/avalon-m1 , agent collectives start slowly due to co-ordination overhead, but eventually pull ahead of single-agent baselines once they pass the initial period. At a matched-token budget, a single-agent baseline takes 4x as long compared to agent collectives. On long-horizon work as measured by EdgeBench, an 8-hire organisation overtakes the single-agent baseline at around 2.4 hours of derived latency. Latency per benchmark, and progress on EdgeBench over time EdgeBench cumulative reward over time Outstanding problems We are training agents in villages but deploying them in cities Latency pressure combined with the right communicative primitives gives us a basic recipe for training agent collectives. However, it is still the first innings for this new research domain, and there are many problems that make scaling collectives difficult. These include: The village problem. There may be a large gap between the size of collectives during training and deployment. For example, in our initial Avalon-M1 model https://www.gr.inc/releases/avalon-m1 , we trained organisations with at most 9 agents. But swarms are now deployed with hundreds or thousands of agents.<sup 12 note-12 , 1 note-1 </sup In effect, we are training collectives in villages but deploying them in cities. We do not know what generalises from small to large scales, and also what emerges at test-time once we use much larger collectives. The uneasy triangle. A flat, collective advantage is a coarse learning signal. Long horizons and shared rewards across multiple agents make credit assignment harder, while training across many environments leaves fewer examples with which to estimate useful updates for each task family. We were able to make the simple approach work at a small scale, but it is unclear how this will scale to much larger collectives with longer episodes. The uneasy triangle of RLCR One advantage over very many tokens Episodes last hours and sometimes millions of tokens, so a flat advantage is applied to many tokens that did not affect the outcome. Building shared environments. In our current training recipe, different rollouts are isolated. But more open-ended settings could allow rollouts to share their work with future generations of agents, e.g. by publishing a paper or code library. It is not clear how to reward rollouts for contributing to a public good that does not directly benefit them in the current episode. Under a per-episode return, a rollout receives no credit for benefits that its contribution creates only in later episodes. Conclusion The reasoning paradigm showed that reinforcement learning could incentivise a model to use additional inference-time compute productively. Collective inference-time scaling extends this principle to how compute is organised across agents. When parallel work produces a result sooner, the training setup should give reduced latency a benefit, and in turn incentivise agents to co-ordinate by communicating. We showed an early example of how to incentivise such behaviour with Avalon-M1 https://www.gr.inc/releases/avalon-m1 . Many outstanding problems remain for this new field of research, including generalising from small-scale collectives at training-time to large-scale collectives at test-time, better credit assignment, and learning more advanced forms of collective behaviour at training-time. You can read the Avalon-M1 technical report https://www.gr.inc/avalonm1.pdf for more details on how we approached an initial version of this problem. We will publish the full paper in the coming weeks once training finishes. Citation Please cite RLCR and Avalon-M1 as: @misc{generalreasoning2026avalonm1, title={Avalon-M1: Reinforcement Learning from Collective Rewards}, author={Iliyan Zarov and Henry Course and Anthony Hartshorn and Ran Achiron and Louis Roche and Akshit Sinha and Chengxi Taylor and Ross Taylor}, year={2026}, publisher={General Reasoning}, url={https://gr.inc/avalonm1.pdf}, } References 1. OpenAI. On the Navier–Stokes Millennium Prize Problem. September 2026. https://openai.com/index/navier-stokes-solution/ 2. K. Kakaes. AI has solved one of math’s $1 million Millennium Prize Problems. Quanta Magazine, 8 September 2026. https://www.quantamagazine.org/ai-has-solved-one-of-maths-1-million-millennium-prize-problems-20260908/ 3. OpenAI. The Hugging Face incident and the road ahead. 2026. https://openai.com/index/hugging-face-incident-and-the-road-ahead/ 4. METR. Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident. August 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/ 5. General Reasoning. Avalon-M1: Reinforcement learning from collective rewards. Technical report, October 2026. https://www.gr.inc/avalonm1.pdf 6. OpenAI. Learning to reason with LLMs. September 2024. https://openai.com/index/learning-to-reason-with-llms/ 7. DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948, 2025. https://arxiv.org/abs/2501.12948 8. J. Hu, Y. Zhang, Q. Han, D. Jiang, X. Zhang and H.-Y. Shum. Open-Reasoner-Zero: An open source approach to scaling up reinforcement learning on the base model. arXiv:2503.24290, 2025. https://arxiv.org/abs/2503.24290 9. DeepSeek-AI. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv:2609.19969, 2026. https://arxiv.org/abs/2609.19969 10. V. Xiang, C. Blagden, R. Rafailov, N. Lile, S. Truong, C. Finn and N. Haber. Just enough thinking: Efficient reasoning with adaptive length penalties reinforcement learning. arXiv:2506.05256, 2025. https://arxiv.org/abs/2506.05256 11. Kimi Team. Kimi K2.5: Visual agentic intelligence. arXiv:2602.02276, 2026. https://arxiv.org/abs/2602.02276 12. Kimi Team. Kimi K2.6. Moonshot AI release note and model card, April 2026. https://www.kimi.ai/blog/kimi-k2-6