You Don't Need to Choose Between a Gateway and an Agent Framework

Swarm, a pure-Rust runtime, offers a unified platform that combines a gateway and an agent orchestrator, allowing teams to start with basic routing and progressively enable advanced features like multi-provider fallbacks, stateful sessions, MCP tool support, and multi-agent DAGs without rewriting their architecture. The project aims to eliminate the false dichotomy between gateways and agent frameworks in AI infrastructure.

When I first published Swarm https://github.com/fcn06/swarm on GitHub, most questions weren't about Rust or MCP. They were about timing and categorization: "We just need a lightweight gateway for multi-provider routing; agents feel like overkill." "We already run an orchestration framework; why would we replace our proxy?" This reaction highlights a false dichotomy currently plaguing the AI infrastructure ecosystem: the assumption that a gateway and an agent orchestrator must be two completely different products. In practice, teams rarely wake up needing full-blown multi-agent autonomous swarms on Day 1. But when they start with a standalone proxy, they inevitably hit a wall — patching together Python microservices, external vector state stores, MCP bridges, and ad-hoc eval scripts. Every evolution requires a rewrite. The core premise of Swarm is different: a single, pure-Rust runtime where you don't choose between a gateway and an orchestrator — you simply choose which capabilities to turn on. Most engineering teams evolve their LLM stack along a predictable trajectory: Rung 1: OpenAI-Compatible Gateway Drop-in replacement for hardcoded SDKs └── Rung 2: Multi-Provider Fallbacks Groq, Gemini, Ollama, vLLM via TOML └── Rung 3: Stateful Sessions Previous response chaining & context └── Rung 4: Native MCP Tools SSE + Streamable HTTP tool execution └── Rung 5: Multi-Agent DAGs Planner + Executor + Specialists └── Rung 6: Built-in Evals LLM-as-a-Judge & policy gates You can stop at any rung and have a lean, production-grade binary. When you're ready for the next level, you change a configuration flag — not your architectural foundation. If your immediate goal is simply eliminating hardcoded API keys and single-vendor SDK locks, Swarm acts as an OpenAI-compatible drop-in front door with sub-millisecond native routing overhead. Spin up the gateway in seconds ./kickstart/gateway kickstart/01 launch gateway.sh curl -X POST http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "openai/gpt-4o-mini", "messages": {"role": "user", "content": "Explain progressive disclosure in software."} }' You get instant OpenAI compatibility. No agent overhead, no background worker queues, no forced abstractions. When rate limits hit or you need cost-effective fallbacks across cloud and local runtimes Groq, Anthropic, Gemini, Ollama, vLLM, llama.cpp , routing is declared cleanly in config.toml : providers.groq api url = "https://api.groq.com/openai/v1/chat/completions" weight = 80 providers.local vllm api url = "http://localhost:8000/v1/chat/completions" recommended models = "meta-llama/Llama-3.3-70B-Instruct" Your applications continue calling the same /v1/chat/completions endpoint. Failover, load distribution, and local-inference routing happen invisibly inside the runtime. /v1/responses Multi-turn chat state is where teams often bolt on an external Redis or PostgreSQL session manager. Swarm provides explicit turn-by-turn state management natively through /v1/responses using previous response id chaining: curl -X POST http://localhost:8080/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "groq/llama-3.3-70b-versatile", "input": "Calculate the Q3 cloud infrastructure spend.", "previous response id": "resp 01JMW892KPA7XYZ" }' State is managed by the runtime, eliminating client-side conversation bloat while keeping state inspection simple and deterministic. When your model needs real-world context — database schemas, filesystem access, API calls — you shouldn't have to migrate to a heavy agent framework just to call tools. Swarm natively supports MCP over both SSE and streamable HTTP directly inside the gateway layer: mcp servers.postgres db transport = "sse" url = "http://localhost:3001/sse" mcp servers.git tools transport = "http" url = "http://localhost:3002/mcp" Tool discovery, argument validation, and streaming tool execution run within the same engine that routes your completions. When single-prompt loops cannot solve compound tasks, Swarm activates its autonomous orchestration engine: User Intent │ ▼ Planner ──► Builds Execution DAG Dependencies & Concurrency │ ▼ Executor ──► Dispatches tasks across Domain Specialists │ ├── Specialist A Data Analyst + Postgres MCP └── Specialist B Report Writer + File MCP │ ▼ Unified Response Why this matters: Rung 5 reuses the identical provider configurations, fallback pools, state engine, and MCP tool connectors established in Rungs 1–4. There is no secondary agent daemon or translation bridge. The final rung is the one most gateways and agent frameworks omit entirely: closing the loop on quality. Instead of exporting logs to an external SaaS pipeline, Swarm embeds an LLM-as-a-Judge loop. It scores intermediate DAG outputs, validates MCP tool results against deterministic schemas, and flags hallucinated responses before they reach client applications: curl -X POST http://localhost:8080/v1/eval/judge \ -H "Content-Type: application/json" \ -d '{ "response id": "resp 01JMW892KPA7XYZ", "criteria": "correctness", "grounding", "conciseness" , "judge model": "openai/gpt-4o" }' This foundational layer enables our upcoming roadmap items: policy-based dynamic routing, durable state checkpoints, and human-in-the-loop validation gates. The individual capabilities of Swarm — gateway proxying, MCP tool invocation, DAG planning, automated evaluation — exist across different open-source projects. What is rare is finding them integrated into a single, zero-dependency, memory-safe binary where adopting multi-agent orchestration doesn't invalidate the proxy architecture you set up on Day 1. The architectural bet of Swarm is simple: the tools you choose when you only need a gateway should never become technical debt the day you need agents. If you're currently scaling your LLM infrastructure: Check out the project and try the kickstart scripts on GitHub: github.com/fcn06/swarm https://github.com/fcn06/swarm