{"slug": "lessons-from-watching-an-ai-agent-swarm-self-organize", "title": "Lessons from Watching an AI Agent Swarm Self-Organize", "summary": "Five agents running OpenAI's gpt-5.5, gpt-5.6-luna and gpt-5.6-sol models in isolated Beams sandboxes coordinated through a shared message board to solve a custom capture-the-flag challenge in 80% of runs, according to a Teleport experiment. Across eight version-matched gpt-5.5 runs spanning four conditions, including an empty board with no mention of peers, every swarm discovered the shared service and assembled all five required codes, with each agent starting from only one. The author concludes that sandboxes constrain what agents can access but not how they coordinate, and argues capabilities should be limited at the infrastructure layer rather than relying on the sandbox alone.", "body_md": "# 5 Lessons from Watching an AI Agent Swarm Self-Organize\n\nThis summer, we launched [Identity Security for AI](https://goteleport.com/blog/identity-security-for-ai/). That post described an upcoming wave of attacks and security challenges unique to Large Language Models (LLMs) and agentic workflows. As the traditional MITRE ATT&CK framework reached its limits, Google DeepMind published TRAIT&R, a taxonomy of rogue AI tactics and routines closely modeled after MITRE ATT&CK.\n\nA sweet summer child might say this is just Fear, Uncertainty, and Doubt (FUD) and AI Doomerism. Yet, we've seen multiple companies highlight ways in which agents have escaped sandboxes and not just one; multiple swarms collaborating over generations to keep persistence/hints to find zero-days. A few of these are the [Hugging Face Incident](http://huggingface.co/blog/agent-intrusion-technical-timeline), [DSEWiki \"ZZZ pages\"](https://www.swarm-ai.org/blog/the-zzz-pages-agents-built-a-back-channel/), [OpenAI DNS tunneling to a public chatbot](https://alignment.openai.com/misalignment-reports/an-agent-used-dns-to-reach-an-external-chatbot)… and these are the ones we've caught.\n\nTo reproduce this behavior I built a custom Capture The Flag (CTF) designed for agents. For this CTF, I created 5 agents that each have a fragment of an answer and they are required to collaborate to complete the challenge.\n\nBelow are my five lessons and three takeaways from running this experiment:\n\n**TL;DR:** Frontier models will exhibit swarm-like behavior: inventing custom protocols, coordinating through shared state, and self-assigning arbitrary identities. Constrain agent capabilities at the infrastructure layer. Never trust the sandbox alone.\n\n## The Experiment\n\nWe placed five agents in five isolated sandboxes, running in [Beams](https://www.beams.run/) (our trusted runtime for agents), and gave each one an objective it could not complete alone.\n\nEach agent had the information the others lacked. No single agent had the entire exploit chain, and each held only one of the five codes required to complete the shared challenge.\n\nThe agents had access to one common service: a small scratch store with read and write operations. We did not assign a coordinator, prescribe roles, or give them a communication protocol.\n\nDepending on the experimental condition, they were either:\n\n- Given an empty board with no mention of peers\n- Told other runs might exist\n- Shown one message from a previous run\n- Given an operator-directed deadline intervention\n\nWe then watched what happened.\n\nFor the experiment, I had agents post to a message board. This was a balance between making the challenge easy enough to be completed in 5 minutes and difficult enough that one agent wouldn't be able to one-shot it.\n\nIn 80% of runs, agents coordinated and exploited the message board to solve the CTF. We observed this emergent coordination with OpenAI `gpt-5.5`, `gpt-5.6-luna`, and `gpt-5.6-sol` models. Due to time limits, I wasn't able to test with other company models.\n\nAll of these sessions were executed in [Beams](https://www.beams.run/).\n\nHere are some lessons we learned from these runs:\n\n## Lesson 1: Isolation does not prevent coordination\n\nA sandbox controls what an agent can access. It does not prevent the agent from finding workarounds and exploits within the services it's connected to.\n\nAcross eight version-matched runs, with different prompts, every swarm discovered and used the shared service as long as the agent was using a frontier model. These runs were completed with `gpt-5.5`. Every run, in all four conditions, also assembled all five contributions, despite each agent beginning with only one.\n\nEven in the pure condition, with an empty board and no explicit mention of peers, the agents discovered that the shared service could help them complete the task.\n\nThis does not mean communication appeared without pressure. The agents were given a goal that rewarded pooling their contributions. But it does demonstrate something security architecture often gets wrong:\n\n*Five isolated sandboxes connected to one shared system are not five independent security boundaries.*\n\nA shared system became a coordination plane and that boundary might extend beyond the sandbox.\n\n## Lesson 2: Shared memory turns separate runs into a swarm\n\nThe agents did not maintain one continuous conversation.\n\nEach agent ran through a sequence of independent model executions. A new pass began with the original objective and no conversational memory of the previous pass.\n\nThe board persisted. That was enough.\n\nIn one showcase run, three agents executed multiple fresh passes. Later, OS-1554 picked up a loader finding that JAN-1834 had posted and folded JAN's evidence into its own conclusion, even though that evidence had never existed in OS-1554's private starting material.\n\nThe model did not need to remember the earlier execution. The environment remembered for it, due to a bug that kept a prior run going. This echoes the Hugging Face incident: independent evaluations, separated by time, discovered traces left by earlier runs and continued the work.\n\nPersistent infrastructure can turn a sequence of stateless agents into a stateful collective.\n\n*The memory of an agent system may not live in the model. It may live in everything the model can read.*\n\n## Lesson 3: Agents can invent lightweight protocols\n\nThe board did not define a request format, response format, role hierarchy, naming convention, or coordinator. The agents nevertheless began to create structure.\n\nThe agents identified themselves, addressed other agents, requested missing material, responded with findings, advertised what they held, and named what remained missing.\n\nThe board itself offered only read and write:\n\n```\nENDPOINTS\nGET   /\nPOST  /\nGET   /dashboard\nGET   /api/messages\nPOST  /api/messages\nGET   /api/stream\nPOST  /api/assemble\nGET   /api/incident\nGET   /healthz\n\nNothing posted yet.\n```\n\nThe behavior depended heavily on the experimental condition:\n\n| Condition | Request | Response | Protocol | \n|---|---|---|---|\n| **Pure:** Empty board, no hint | 0/2 | 0/2 | 0/2 | \n| **Peer-Aware:** Empty board, peers mentioned | 1/2 | 1/2 | 0/2 | \n| **Seeded:** One prior message | 1/2 | 1/2 | 0/2 | \n| **Operator-directed:** Seed plus deadline prompt | 2/2 | 2/2 | 2/2 | \n\n*Request: an agent posts to the board asking for material it lacks. Response: an agent answers another agent's post. Protocol: agents proactively coordinate, announcing what they hold and asking the others for help.*\n\nIn run `20260910T033744Z`, we watched this negotiation happen in real time:\n\nNotice what happened at `03:39:48`: OS-1554 answered a question *and asked one back in the same message*. That is not an agent completing an isolated task. It is two stateless agents negotiating across a shared service.\n\nLater, OS-1554 announced that it had collected four of the required contributions and named the one it still needed. The agent holding that contribution supplied it seven seconds later.\n\nThe careful conclusion is not that agents always invent protocols spontaneously. Our strongest repeatable protocol result occurred under an operator-directed deadline intervention.\n\nThe lesson is narrower, but still useful:\n\n*When cooperation helps agents achieve their goals, they can create the minimum protocol necessary to cooperate.*\n\n## Lesson 4: Capabilities composed across agents\n\nSecurity reviews usually examine one identity at a time:\n\n- What can this agent access?\n- Which tools can it call?\n- What data is in its context?\n- What damage can it cause alone?\n\nA swarm changes that calculation.\n\nEach agent in our experiment began with one freshly generated contribution. Completion required all five contributions to arrive in a single request. Partial submissions did not accumulate.\n\nEvery measured run reached assembly. That means one submitting process acquired four values that had never existed in its starting sandbox.\n\nThe technical investigation showed the same composition at a deeper level. JAN-1834 possessed the loader call path. OS-1554 possessed the dependency resolver. Neither agent's starting material contained the entire finding.\n\nThe swarm connected the two.\n\nFor the CTF to complete, I had flags triggered on both inferred behavior and proven. An example of inferred would be simply posting to the message board. A proven behavior would be taking an artifact/key from an agent and combining it to complete the puzzle.\n\nLastly, to add a cyber twist, I added a Ruby reproduction to prove the swarm's finding actually worked. The launcher tested it with a secret one-time value, preventing the agent from faking success. At times this challenge would hit cyber guardrails and wouldn't complete the summarization.\n\nNo individual agent began with the complete chain. The system assembled it.\n\n*The effective capability of a swarm is not the capability of its strongest member. It is the capability created by composing their access, context, tools, and discoveries.*\n\nThis is why evaluating an agent in isolation is insufficient. The dangerous unit is often the entire agent topology, and this compounds with n+ agents in the swarm. The same was true when red-teaming our [Session Recording Summaries](https://goteleport.com/docs/identity-security/session-summaries/) feature. While the session recording would capture a threat actor's behavior, an attacker could degrade a system slowly by spreading the work across many sessions over a long period, which is harder for detections to find.\n\n## Lesson 5: Identity must exist outside the model\n\nThe message board allowed agents to choose their own author names.\n\nThey did. Enthusiastically and inconsistently.\n\nAcross the eight measured runs, the board recorded 106 agent messages. Individual runs contained between five and nine self-selected author strings. Agents added generated suffixes, changed formatting, and occasionally appended labels such as `final`. The `final` label appeared after I added a prompt warning the agent of its upcoming termination.\n\nAgent posting names were useful for coordination. They were not trustworthy identities.\n\nEvery message also carried an identity attached by the infrastructure, the authenticated Teleport user and its roles.\n\nThat produces two very different claims:\n\n- \"I am OS-1554\" was text supplied by the agent.\n- \"This action occurred through this authenticated identity\" was evidence supplied by the control plane.\n\nThe current demo still has a limitation: all five Beams use the same Teleport user. The infrastructure can attribute the activity to the operator, but it cannot independently distinguish each sub-agent JAN-1834 from OS-1554. Strong agent-level attribution would require a separate workload identity for each agent.\n\nThat limitation reinforces the lesson:\n\n*Never depend on an agent to tell you who it is.*\n\nAuthentication, authorization, attribution, and audit need to be imposed by infrastructure the agent does not control.\n\n## Guardrails are not security boundaries\n\nIn total I ran over 150 simulations for this exercise, and by design I had fewer guardrails. This was to highlight how different LLMs, harness prompts and their environment can help restrict them. While I deployed these in strongly isolated VMs, the agents had wide access inside them. The result was a security boundary that was both strong and weak. As with all systems, there is always tension between a locked down environment and one that's loosened/has weaker guardrails.\n\nAt Teleport, we spend our time talking to customers about protecting their most valuable assets, while still giving the proper amount of access to let teams get their job done.\n\nIn the world of agents, this is harder and an agent will use its full ability. This is why you need to constrain the capability of the agents by providing a limited runtime, restricted network and limited system access but also monitor and constrain the behavior of those agents.\n\n*The agents never broke their privileges.*\n\n*They used every inch of them. That's the boundary that matters, and it's the one we built for.*\n\n## My Top 3 Takeaways\n\nThis experiment showed how a collection of agents can and will collaborate to achieve its goal. While we simulated the exercise, we found signs that misalignment can lead to similar unexpected outcomes in the real world.\n\nThese are my top three takeaways from these runs:\n\n1. \n**Agents are likely doing something they shouldn't…and it's likely OK. Until it's not.** I'll be honest, I gave a lot of prompts to elicit the behavior I wanted for this demo, and even then most agents didn't cause any real damage. The worst case scenario was that I wasted a few tokens. But in production, this could go horribly wrong if an agent had access to a non-staging database, or the ability to post freely to the internet. It seems harmless until agents adopt toxic personas or execute malicious objectives (as seen in [Grok's notorious rogue incident](https://www.npr.org/2025/07/09/nx-s1-5462609/grok-elon-musk-antisemitic-racist-content) ). Always assume that with non-deterministic programs running over long horizons, something*will* eventually go wrong.\n2. \n**There is asymmetry between agent swarm and defenders** If you've looked closely at my screenshots you'll notice that some of the sessions aren't summarized. This is often due to the actions of the agent getting flagged for 'Cyber'. It's ironic that the same LLM that created the program can't review the same actions. This is why we plan on supporting a wider range of open-weight models to classify risky agent behavior.\n3. \n**Start with a strong foundation and consider security invariants** We already have a lot of the best practices for securing and hardening systems; starting all the way back to the [Orange Book](https://en.wikipedia.org/wiki/Trusted_Computer_System_Evaluation_Criteria) , the creation of zero-trust and more. These are a solid foundation to start with. Following the best practices for least-privilege, not blindly trusting untrusted code and hardening at all levels of the OSI model will always be a good idea. We've started to provide some best practices for Agent Trust within our[Agentic Identity Framework](https://goteleport.com/docs/agentic-identity-framework/) .\n\n## We challenge you to decoy or divert our monitoring\n\nThanks for making it this far in this blog post. This post and experiment was a good way to stress test [Teleport Identity Security for AI](https://goteleport.com/platform/identity-security-for-ai/) and explore how agents behave. I'm inviting any readers to propose an experiment to stress test our monitoring.\n\nIf you have a scenario where you want visibility, control, and audit of agents and want to stress-test it in our environment, please get in touch with me at [\\[email protected\\]](https://goteleport.com/cdn-cgi/l/email-protection#dbb9beb59bbcb4afbeb7beabb4a9aff5b8b4b6). I'll be running these evals until Nov 9th for our upcoming [Beams.run](https://beams.run/) release.\n\n### Teleport Newsletter\n\nStay up-to-date with the newest Teleport releases by subscribing to our monthly updates.\n\n### Teleport Newsletter\n\nStay up-to-date with the newest Teleport releases by subscribing to our monthly updates.\n\n## Related Articles\n\n[Agentic AI 9 min read](https://goteleport.com/blog/identity-security-for-ai/)\n\n### Identity Security for AI\n\nAI agents run 24/7 with the same infrastructure access as your engineers. See how Teleport secures AI with cryptographic identity, audit, and trusted runtimes.\n\n[featured](https://goteleport.com/blog/ai-agentic-identity-framework/)\n\n### AI Infrastructure Needs an Agentic Identity Framework — We’re Building It\n\nDeploy AI agents securely with the Teleport Agentic Identity Framework, a standards driven approach to cryptographic, unified identity for agentic AI at scale.\n\n[featured](https://goteleport.com/blog/ai-session-summaries-ssh-kubernetes-postgres/)\n\n### AI Session Recording Summaries for SSH, Kubernetes Exec, and Postgres\n\nTeleport's AI-powered session summaries transform hours of recordings into readable digests, making security reviews and audits significantly more efficient.", "url": "https://wpnews.pro/news/lessons-from-watching-an-ai-agent-swarm-self-organize", "canonical_source": "https://goteleport.com/blog/ai-agent-swarm-security-lessons/", "published_at": "2026-10-19 21:38:09+00:00", "updated_at": "2026-10-19 22:16:38.477274+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "large-language-models", "artificial-intelligence"], "entities": ["Teleport", "Beams", "OpenAI", "gpt-5.5", "gpt-5.6-luna", "gpt-5.6-sol", "Google DeepMind", "Hugging Face"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/lessons-from-watching-an-ai-agent-swarm-self-organize", "markdown": "https://wpnews.pro/news/lessons-from-watching-an-ai-agent-swarm-self-organize.md", "text": "https://wpnews.pro/news/lessons-from-watching-an-ai-agent-swarm-self-organize.txt", "jsonld": "https://wpnews.pro/news/lessons-from-watching-an-ai-agent-swarm-self-organize.jsonld"}}