While experimenting with multi-agent systems, I kept ending up with two separate pieces of infrastructure: an orchestration layer for agents and tools, and a gateway layer for routing LLM requests.
I wanted both to share the same runtime, provider abstractions, state management, and protocol contracts.
So I built ** Swarm**, an open-source AI orchestration framework and model gateway written in Rust.
Many AI stacks end up separating these concerns: a dedicated proxy for lightweight routing and a separate orchestrator for more complex reasoning. Swarm unifies both patterns around a single high-performance Tokio runtime.
+--------------------------------------------------------------------------------------------------+
| SWARM MODES |
+--------------------------------------------------------------------------------------------------+
| |
| MODE 1: MULTI-AGENT & MCP ORCHESTRATION MODE 2: MODEL GATEWAY SERVER |
| (kickstart/multi_agent_orchestration_kickstart/) (kickstart/gateway_kickstart/) |
| |
| • Planner Agent (Dynamic plan generation) • POST /v1/chat/completions (OpenAI) |
| • Executor Agent (Workflow DAG execution) • POST /v1/responses (Open Responses) |
| • Domain Specialists with MCP Tool integration • Stateful multi-turn chaining |
| • Discovery & Memory services • Multi-provider (Groq, Gemini, OpenAI, |
| • Evaluation & Judge Service Ollama / vLLM / local endpoints) |
| • Resilient OAuth2 / JWT authentication • High-throughput lock-free cache |
| |
+--------------------------------------------------------------------------------------------------+
The key idea: Swarm can run as a full agent orchestration stack or as a standalone LLM gateway without requiring two unrelated frameworks.
Coordinating multiple agents becomes much easier when service boundaries and message contracts are explicit.
Mode 1 splits responsibilities across decoupled, specialized services:
Inter-agent communication relies on type-safe agent-to-agent (A2A) message contracts, catching many contract and integration errors during development and compilation.
User Request
↓
Planner
↓
Execution DAG
↓
Executor
↓
Weather Agent
↓
MCP Weather Tool
↓
Evaluation
↓
Final Response
Mode 2 exposes an OpenAI-compatible gateway for client applications, developer tools, and automated pipelines.
POST /v1/chat/completions
):POST /v1/responses
):previous_response_id
references.
[server]
bind_address = "0.0.0.0:8080"
log_level = "info"
[models]
default_model = "openai/gpt-oss-20b"
[providers.groq]
api_url = "https://api.groq.com/openai/v1/chat/completions"
[providers.google]
api_url = "https://generativelanguage.googleapis.com/v1beta/models"
[providers.custom]
api_url = "http://localhost:11434/v1/chat/completions"
recommended_models = ["llama3.2:latest", "mistral:latest", "deepseek-r1:8b"]
Rust gives Swarm a few useful properties for orchestration and gateway workloads:
DashMap
and Arc
-based stores allow shared state across concurrent requests without a global application lock.You can test either mode locally in minutes:
git clone https://github.com/fcn06/swarm.git
cd swarm
cp .env.example .env
./kickstart/gateway_kickstart/01_launch_gateway.sh
curl -X POST http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "openai/gpt-oss-20b",
"messages": [{"role": "user", "content": "Explain Swarm architecture in 2 sentences."}]
}'
./kickstart/multi_agent_orchestration_kickstart/01_launch_all.sh
./kickstart/multi_agent_orchestration_kickstart/02_test_weather_query.sh "What is the current weather in Boston ?"
Swarm is fully open-source under the Apache-2.0 license. We rely on and contribute back to the emerging Rust AI ecosystem, including the official MCP Rust SDK and A2A Protocol.
I'm especially interested in feedback from people running agent systems or LLM gateways in production:
Would you rather deploy the orchestration and gateway as one runtime, or keep them completely separate?
If you try Swarm, I'd also love feedback on the MCP runtime, gateway compatibility, and APIs.