# Building Multi-Agent Systems That Actually Scale: Lessons from Hermes, LobeHub, and the 2025 AI Agent Explosion > Source: > Published: 2026-08-16 06:01:34+00:00 *Originally published on tamiz.pro.* The AI agent landscape shifted dramatically in 2025. What began as single-agent chat interfaces exploded into multi-agent ecosystems where dozens or hundreds of specialized agents coordinate, debate, and execute complex workflows. Hermes and LobeHub emerged as two distinctive approaches to this problem—neither a toy demo nor an enterprise suite—and their architectural decisions reveal what it actually takes to build systems that scale beyond a handful of concurrent agents. This is not a survey of agent frameworks. It's an engineering post-mortem on the hard problems that appear when you stop treating agents as isolated LLM calls and start thinking about them as networked services. A single agent calling an LLM is a well-understood pattern. You send a prompt, you get a response, you handle latency and token budgets. The mental model is straightforward because the topology is trivial: one request, one agent, one model call. Multi-agent systems introduce three compounding difficulties: The naive approach—fire all agents in parallel, let them talk through a shared message bus, aggregate results—works until you try to run it at scale. Then you hit resource contention, unbounded fan-out, and prompt injection attacks that flow through trust boundaries between agents. Hermes and LobeHub solved this differently. Understanding both approaches is the best way to internalize the design space. Hermes approaches multi-agent coordination as a **message-passing system with typed interaction protocols**. Its key insight is that most agent-to-agent communication follows predictable patterns—subtask delegation, result synthesis, conflict resolution—and these patterns should be explicit, not emergent. ``` ┌─────────────────────────────────────────────────┐ │ User Request │ └──────────────────────┬──────────────────────────┘ ▼ ┌────────────────┐ │ Orchestrator │ ← Static routing table + dynamic load balancing └────────┬───────┘ ▼ ┌──────────────────────────────┐ │ Message Router │ ← Typed message queues per agent group └────────┬─────────────┬───────┘ ▼ ▼ ┌────────────────┐ ┌────────────────┐ │ Worker Pool │ │ Specialist │ │ (parallel) │ │ (sequential) │ └────────┬───────┘ └────────┬───────┘ ▼ ▼ ┌────────────────────────────────────┐ │ Result Aggregator │ ← Deterministic merge + LLM reconciliation └────────────────────────────────────┘ ``` Hermes separates agents into two tiers: The critical design choice is that workers never talk to workers directly. All inter-agent communication flows through the orchestrator, which enforces a **directed acyclic communication graph**. This prevents the combinatorial explosion of agent-to-agent messaging and makes the execution plan auditable. Hermes introduces a minimal schema for agent messages: ``` // Core message types in Hermes interface AgentMessage { id: string; type: 'delegation' | 'result' | 'conflict' | 'escalation'; sender: AgentId; recipient: AgentId | 'orchestrator'; payload: AgentPayload; contextRef?: string; // Reference to shared context blob ttl: number; // Time-to-live in seconds } interface AgentPayload { task?: string; result?: AgentResult; error?: AgentError; vote?: { agentId: AgentId; confidence: number; reasoning: string }; } interface AgentResult { output: string; tools_used: string[]; tokens_consumed: number; confidence: number; citations?: SourceReference[]; } ``` This might look like unnecessary boilerplate next to free-form agent conversations, but it's the difference between a system you can monitor and debug versus one you cannot. When an agent call fails at 3 AM, you want structured error propagation, not a black-box reasoning trace you have to parse from prose. The cost model is where Hermes gets interesting. Instead of letting agents burn tokens freely, it implements a **budget-aware routing** layer: ``` python class BudgetAwareRouter: def __init__(self, agent_registry, cost_tracker): self.agents = agent_registry self.cost = cost_tracker self.default_budget_per_request = 5000 # tokens self.max_concurrent_agents = 8 async def route(self, user_request, context): plan = self._decompose_task(user_request) cost_estimate = self._estimate_cost(plan) if cost_estimate > self.default_budget_per_request: # Trigger agent compression: merge low-value agents plan = self._compress(plan) return await self._execute_plan(plan) ``` The router estimates token costs *before* execution by analyzing the task decomposition. If a plan exceeds the budget, it compresses the agent graph—merging redundant specialists or falling back to cheaper models for lower-priority steps. This is not a hard limit; it's a heuristic optimization that prevents runaway costs on complex requests. LobeHub takes a fundamentally different approach. Rather than enforcing a rigid top-down orchestration, it treats agent coordination as a **peer-to-peer mesh with gossip-style consensus**. ``` ┌─────────┐ │ Agent A │◄──────────────────────────────────┐ └────┬────┘ │ │ gossip │ shared state ┌────▼────┐ ┌─────────────┐ │ Agent B │◄─────────────────────────────│ State Bus │ └────┬────┘ └──────┬──────┘ │ │ ┌────▼────┐ │ │ Agent C │◄─────────────────────────────────────┘ └─────────┘ All agents read/write to a shared state bus. Consensus emerges through voting rounds. ``` LobeHub's design philosophy is that complex problems benefit from **diverse, concurrent reasoning** rather than sequential decomposition. Multiple agents work on the same problem independently, then reach consensus through a voting mechanism. ``` interface ConsensusRound { roundId: string; question: string; participants: AgentId[]; deadline: number; // Unix timestamp quorumSize: number; strategy: 'majority' | 'weighted' | 'unanimous'; votes: Vote[]; result?: ConsensusResult; } interface Vote { agentId: AgentId; position: string; confidence: number; reasoning: string; timestamp: number; } interface ConsensusResult { agreedPosition: string; confidence: number; dissentingViews: DissentingView[]; roundId: string; converged: boolean; } ``` The consensus mechanism is the heart of LobeHub. When agents disagree—and they will, because LLMs are non-deterministic—the system doesn't default to a simple majority. It runs **weighted voting** where agents with higher historical accuracy on similar tasks get more influence. ``` python class WeightedConsensusEngine: def __init__(self, agent_reputation_store): self.reputation = agent_reputation_store self.confidence_threshold = 0.75 self.max_rounds = 3 async def reach_consensus(self, round: ConsensusRound) -> ConsensusResult: votes = await self.collect_votes(round) if not self._has_quorum(votes, round.quorumSize): return await self.expand_participants(round) weighted = self._apply_weights(votes) agreed = self._find_agreement(weighted) if agreed.confidence < self.confidence_threshold and round.round_num < self.max_rounds: # Trigger a refinement round with targeted follow-up questions return await self.run_refinement_round(round, agreed) return agreed ``` This is essentially a **distributed reasoning protocol**—borrowing from consensus algorithms like Raft and Paxos, but adapted for probabilistic, non-deterministic participants. The refinement rounds are particularly clever: instead of just re-voting, the system identifies specific disagreement points and asks agents to address them directly. **Strengths:** **Weaknesses:** Three shifts in 2024–2025 made multi-agent systems viable at production scale: Early agent frameworks struggled with parsing LLM outputs reliably. Function calling was inconsistent, JSON extraction failed silently, and error handling was an afterthought. By 2025, the major providers (OpenAI, Anthropic, Google) had matured their structured output APIs to the point where deterministic parsing became tractable. This is the single most important infrastructure development for multi-agent systems—if you can't reliably parse an agent's response, you can't build a protocol around it. 200K+ context windows mean agents can share substantial state without constant round-trips. Hermes leverages this with a **shared context blob** pattern: instead of re-explaining the problem to every agent, a compressed context representation is passed along, and agents operate on a shared understanding. This reduces both latency and token costs significantly. The early 2024 trend of building "general purpose" AI assistants collapsed under its own weight. Too many agentic loops, too little specialization, too much hallucination. The 2025 winners are systems where each agent has a clearly defined scope, well-specified tool access, and measurable accuracy. This is why both Hermes and LobeHub start with agent specialization as a first principle, not a retrospective cleanup. Despite their architectural differences, Hermes and LobeHub converge on several patterns that appear to be necessary for any production multi-agent system: Every agent needs a stable identity, not just a name. This includes: `idle` → `assigned` → `active` → `completed` / `failed` ``` interface AgentManifest { agentId: string; version: string; capabilities: Capability[]; model: ModelSpec; toolAccess: ToolAccessPolicy; maxContextTokens: number; timeoutMs: number; } interface AgentState { agentId: string; status: 'idle' | 'assigned' | 'active' | 'completed' | 'failed' | 'timeout'; currentTask?: string; sessionHistory: MessageHistory; lastActiveAt: number; errorCount: number; } ``` Both systems implement fallback chains at every layer: You cannot debug what you cannot observe. Both systems treat observability as a first-class concern: ``` # Example: Structured logging for agent lifecycle class AgentTelemetry: def __init__(self): self.tracer = OpenTelemetryTracer("multi-agent-system") self.metrics = PrometheusMetrics("agent_system") async def record_agent_call(self, call: AgentCall) -> None: with self.tracer.start_span("agent.invocation", trace_id=call.trace_id) as span: span.set_attribute("agent.id", call.agent_id) span.set_attribute("agent.model", call.model) span.set_attribute("tokens.input", call.input_tokens) span.set_attribute("tokens.output", call.output_tokens) span.set_attribute("latency_ms", call.latency_ms) span.set_attribute("confidence", call.result.confidence) span.set_attribute("cost_cents", call.estimated_cost_cents) if call.error: span.record_exception(call.error) self.metrics.increment("agent.errors", labels={"agent": call.agent_id}) else: self.metrics.histogram("agent.latency", call.latency_ms, labels={"agent": call.agent_id}) ``` Agents must never share mutable state directly. Each agent operates on an immutable snapshot of the context it needs, and any state changes are communicated through explicit messages. This prevents race conditions, makes debugging tractable, and enables replay of agent interactions for quality assurance. Based on post-mortems from both Hermes and LobeHub deployments, these are the failure modes that appear most frequently: The temptation is to spawn an agent for every subtask, every validation step, every edge case. This creates exponential cost and latency. Both systems enforce **fan-out caps** at the orchestrator level: ``` interface OrchestrationPolicy { maxDepth: number; // Max nesting of agent calls maxFanOut: number; // Max concurrent agents maxTotalAgents: number; // Hard cap per request budgetCapTokens: number; // Token budget budgetCapUSD: number; // Cost cap circuitBreaker: { errorRateThreshold: number; // Trigger if >X% of agents fail timeoutThreshold: number; // Trigger if avg latency >Y ms }; } ``` When agents communicate freely, a malicious user input can propagate through the agent network. If Agent A generates a message to Agent B, and that message contains instructions that Agent B treats as authoritative, you have a prompt injection attack vector. Both systems implement: LLM outputs are probabilistic. Two identical requests to the same agent can produce different results. This makes testing nearly impossible and debugging frustrating. Mitigation strategies: As agents accumulate conversation history, the context window fills up quickly. Both systems use aggressive context compression: Neither Hermes nor LobeHub is universally better. The right architecture depends on your requirements: | Criterion | Hermes (Top-Down) | LobeHub (Peer-to-Peer) | |---|---|---| Latency sensitivity | Better—sequential specialists are fast | Worse—consensus rounds add latency | Cost predictability | Better—bounded execution plans | Worse—dynamic agent counts | Solution quality | Good for well-defined problems | Better for creative/open-ended problems | Debuggability | Excellent—linear traces | Poor—emergent behavior | Scalability | Linear with worker pool size | Super-linear with participant count | Resilience | Orchestrator is a bottleneck | True fault tolerance | Best for | Transactional workflows (support, analytics, coding) | Exploratory workflows (research, design, strategy) | The most promising systems in 2025 combine both approaches. They use Hermes-style orchestration for the high-level task decomposition and LobeHub-style consensus for the reasoning-critical subtasks. The result is a system that is both fast and creative, bounded and exploratory. A typical hybrid flow: This hybrid approach is becoming the default pattern for production multi-agent systems because it captures the strengths of both paradigms while mitigating their weaknesses. If you're planning to build a multi-agent system in 2025, here's what the evidence from Hermes, LobeHub, and related systems suggests you need: **Q: Do I really need a custom multi-agent framework, or can I use LangGraph or AutoGen?** For simple workflows, yes—those frameworks are fine. But they abstract away the scaling concerns that matter in production: cost control, observability, prompt injection defense, and resilience under load. Hermes and LobeHub's approaches show that these concerns require architectural decisions that generic frameworks don't address. If you're building something that needs to run reliably at scale, investing in purpose-built infrastructure pays off. **Q: How do I choose between sequential orchestration and peer-to-peer consensus?** The rule of thumb: use sequential orchestration when the problem has clear sub-components that can be decomposed and solved independently (data processing, code generation, support ticket resolution). Use consensus-based approaches when the problem requires creative synthesis, trade-off analysis, or when multiple valid answers exist and you need to find the best one (strategy planning, research synthesis, design review). Hybrid approaches get you the best of both worlds. **Q: What's the realistic cost per complex multi-agent request?** In production systems observed in 2025, a well-optimized multi-agent request with 5–8 agents typically costs $0.50–$3.00 depending on complexity and model choices. Unoptimized systems can easily exceed $10 per request. The key differentiator is whether you're using budget-aware routing and context compression. If your costs are higher, you're likely spawning too many agents or failing to compress shared context effectively. The 2025 AI agent explosion isn't about making individual agents smarter—it's about building systems where many specialized agents coordinate effectively. Hermes and LobeHub represent two proven points in that design space, and the hybrid architectures they're converging toward suggest the field is still evolving rapidly. The engineers who internalize these patterns now will have a significant advantage as multi-agent systems move from experimental to essential infrastructure.