{"slug": "building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and", "title": "Building Multi-Agent Systems That Actually Scale: Lessons from Hermes, LobeHub, and the 2025 AI Agent Explosion", "summary": "Engineer Tamizhselvan (tamiz.pro) published a technical post-mortem on building scalable multi-agent systems, drawing lessons from Hermes and LobeHub. The article details how Hermes uses typed message-passing protocols and a directed acyclic communication graph to avoid the combinatorial explosion of agent-to-agent messaging, contrasting it with LobeHub's approach. It highlights the engineering challenges of resource contention, unbounded fan-out, and prompt injection attacks in multi-agent architectures.", "body_md": "*Originally published on tamiz.pro.*\n\nThe AI agent landscape shifted dramatically in 2025. What began as single-agent chat interfaces exploded into multi-agent ecosystems where dozens or hundreds of specialized agents coordinate, debate, and execute complex workflows. Hermes and LobeHub emerged as two distinctive approaches to this problem—neither a toy demo nor an enterprise suite—and their architectural decisions reveal what it actually takes to build systems that scale beyond a handful of concurrent agents.\n\nThis is not a survey of agent frameworks. It's an engineering post-mortem on the hard problems that appear when you stop treating agents as isolated LLM calls and start thinking about them as networked services.\n\nA single agent calling an LLM is a well-understood pattern. You send a prompt, you get a response, you handle latency and token budgets. The mental model is straightforward because the topology is trivial: one request, one agent, one model call.\n\nMulti-agent systems introduce three compounding difficulties:\n\nThe naive approach—fire all agents in parallel, let them talk through a shared message bus, aggregate results—works until you try to run it at scale. Then you hit resource contention, unbounded fan-out, and prompt injection attacks that flow through trust boundaries between agents.\n\nHermes and LobeHub solved this differently. Understanding both approaches is the best way to internalize the design space.\n\nHermes approaches multi-agent coordination as a **message-passing system with typed interaction protocols**. Its key insight is that most agent-to-agent communication follows predictable patterns—subtask delegation, result synthesis, conflict resolution—and these patterns should be explicit, not emergent.\n\n```\n┌─────────────────────────────────────────────────┐\n│                   User Request                   │\n└──────────────────────┬──────────────────────────┘\n                       ▼\n              ┌────────────────┐\n              │   Orchestrator  │ ← Static routing table + dynamic load balancing\n              └────────┬───────┘\n                       ▼\n        ┌──────────────────────────────┐\n        │       Message Router         │ ← Typed message queues per agent group\n        └────────┬─────────────┬───────┘\n                 ▼             ▼\n      ┌────────────────┐ ┌────────────────┐\n      │   Worker Pool   │ │   Specialist    │\n      │   (parallel)   │ │   (sequential) │\n      └────────┬───────┘ └────────┬───────┘\n               ▼                   ▼\n      ┌────────────────────────────────────┐\n      │         Result Aggregator           │ ← Deterministic merge + LLM reconciliation\n      └────────────────────────────────────┘\n```\n\nHermes separates agents into two tiers:\n\nThe critical design choice is that workers never talk to workers directly. All inter-agent communication flows through the orchestrator, which enforces a **directed acyclic communication graph**. This prevents the combinatorial explosion of agent-to-agent messaging and makes the execution plan auditable.\n\nHermes introduces a minimal schema for agent messages:\n\n```\n// Core message types in Hermes\ninterface AgentMessage {\n  id: string;\n  type: 'delegation' | 'result' | 'conflict' | 'escalation';\n  sender: AgentId;\n  recipient: AgentId | 'orchestrator';\n  payload: AgentPayload;\n  contextRef?: string;  // Reference to shared context blob\n  ttl: number;          // Time-to-live in seconds\n}\n\ninterface AgentPayload {\n  task?: string;\n  result?: AgentResult;\n  error?: AgentError;\n  vote?: { agentId: AgentId; confidence: number; reasoning: string };\n}\n\ninterface AgentResult {\n  output: string;\n  tools_used: string[];\n  tokens_consumed: number;\n  confidence: number;\n  citations?: SourceReference[];\n}\n```\n\nThis might look like unnecessary boilerplate next to free-form agent conversations, but it's the difference between a system you can monitor and debug versus one you cannot. When an agent call fails at 3 AM, you want structured error propagation, not a black-box reasoning trace you have to parse from prose.\n\nThe cost model is where Hermes gets interesting. Instead of letting agents burn tokens freely, it implements a **budget-aware routing** layer:\n\n``` python\nclass BudgetAwareRouter:\n    def __init__(self, agent_registry, cost_tracker):\n        self.agents = agent_registry\n        self.cost = cost_tracker\n        self.default_budget_per_request = 5000  # tokens\n        self.max_concurrent_agents = 8\n\n    async def route(self, user_request, context):\n        plan = self._decompose_task(user_request)\n        cost_estimate = self._estimate_cost(plan)\n\n        if cost_estimate > self.default_budget_per_request:\n            # Trigger agent compression: merge low-value agents\n            plan = self._compress(plan)\n\n        return await self._execute_plan(plan)\n```\n\nThe router estimates token costs *before* execution by analyzing the task decomposition. If a plan exceeds the budget, it compresses the agent graph—merging redundant specialists or falling back to cheaper models for lower-priority steps. This is not a hard limit; it's a heuristic optimization that prevents runaway costs on complex requests.\n\nLobeHub takes a fundamentally different approach. Rather than enforcing a rigid top-down orchestration, it treats agent coordination as a **peer-to-peer mesh with gossip-style consensus**.\n\n```\n     ┌─────────┐\n     │ Agent A  │◄──────────────────────────────────┐\n     └────┬────┘                                    │\n          │ gossip                                   │ shared state\n     ┌────▼────┐                              ┌─────────────┐\n     │ Agent B  │◄─────────────────────────────│  State Bus   │\n     └────┬────┘                              └──────┬──────┘\n          │                                           │\n     ┌────▼────┐                                      │\n     │ Agent C  │◄─────────────────────────────────────┘\n     └─────────┘\n\n     All agents read/write to a shared state bus.\n     Consensus emerges through voting rounds.\n```\n\nLobeHub's design philosophy is that complex problems benefit from **diverse, concurrent reasoning** rather than sequential decomposition. Multiple agents work on the same problem independently, then reach consensus through a voting mechanism.\n\n```\ninterface ConsensusRound {\n  roundId: string;\n  question: string;\n  participants: AgentId[];\n  deadline: number;       // Unix timestamp\n  quorumSize: number;\n  strategy: 'majority' | 'weighted' | 'unanimous';\n  votes: Vote[];\n  result?: ConsensusResult;\n}\n\ninterface Vote {\n  agentId: AgentId;\n  position: string;\n  confidence: number;\n  reasoning: string;\n  timestamp: number;\n}\n\ninterface ConsensusResult {\n  agreedPosition: string;\n  confidence: number;\n  dissentingViews: DissentingView[];\n  roundId: string;\n  converged: boolean;\n}\n```\n\nThe consensus mechanism is the heart of LobeHub. When agents disagree—and they will, because LLMs are non-deterministic—the system doesn't default to a simple majority. It runs **weighted voting** where agents with higher historical accuracy on similar tasks get more influence.\n\n``` python\nclass WeightedConsensusEngine:\n    def __init__(self, agent_reputation_store):\n        self.reputation = agent_reputation_store\n        self.confidence_threshold = 0.75\n        self.max_rounds = 3\n\n    async def reach_consensus(self, round: ConsensusRound) -> ConsensusResult:\n        votes = await self.collect_votes(round)\n\n        if not self._has_quorum(votes, round.quorumSize):\n            return await self.expand_participants(round)\n\n        weighted = self._apply_weights(votes)\n        agreed = self._find_agreement(weighted)\n\n        if agreed.confidence < self.confidence_threshold and round.round_num < self.max_rounds:\n            # Trigger a refinement round with targeted follow-up questions\n            return await self.run_refinement_round(round, agreed)\n\n        return agreed\n```\n\nThis is essentially a **distributed reasoning protocol**—borrowing from consensus algorithms like Raft and Paxos, but adapted for probabilistic, non-deterministic participants. The refinement rounds are particularly clever: instead of just re-voting, the system identifies specific disagreement points and asks agents to address them directly.\n\n**Strengths:**\n\n**Weaknesses:**\n\nThree shifts in 2024–2025 made multi-agent systems viable at production scale:\n\nEarly agent frameworks struggled with parsing LLM outputs reliably. Function calling was inconsistent, JSON extraction failed silently, and error handling was an afterthought. By 2025, the major providers (OpenAI, Anthropic, Google) had matured their structured output APIs to the point where deterministic parsing became tractable. This is the single most important infrastructure development for multi-agent systems—if you can't reliably parse an agent's response, you can't build a protocol around it.\n\n200K+ context windows mean agents can share substantial state without constant round-trips. Hermes leverages this with a **shared context blob** pattern: instead of re-explaining the problem to every agent, a compressed context representation is passed along, and agents operate on a shared understanding. This reduces both latency and token costs significantly.\n\nThe early 2024 trend of building \"general purpose\" AI assistants collapsed under its own weight. Too many agentic loops, too little specialization, too much hallucination. The 2025 winners are systems where each agent has a clearly defined scope, well-specified tool access, and measurable accuracy. This is why both Hermes and LobeHub start with agent specialization as a first principle, not a retrospective cleanup.\n\nDespite their architectural differences, Hermes and LobeHub converge on several patterns that appear to be necessary for any production multi-agent system:\n\nEvery agent needs a stable identity, not just a name. This includes:\n\n`idle`\n\n→ `assigned`\n\n→ `active`\n\n→ `completed`\n\n/ `failed`\n\n```\ninterface AgentManifest {\n  agentId: string;\n  version: string;\n  capabilities: Capability[];\n  model: ModelSpec;\n  toolAccess: ToolAccessPolicy;\n  maxContextTokens: number;\n  timeoutMs: number;\n}\n\ninterface AgentState {\n  agentId: string;\n  status: 'idle' | 'assigned' | 'active' | 'completed' | 'failed' | 'timeout';\n  currentTask?: string;\n  sessionHistory: MessageHistory;\n  lastActiveAt: number;\n  errorCount: number;\n}\n```\n\nBoth systems implement fallback chains at every layer:\n\nYou cannot debug what you cannot observe. Both systems treat observability as a first-class concern:\n\n```\n# Example: Structured logging for agent lifecycle\nclass AgentTelemetry:\n    def __init__(self):\n        self.tracer = OpenTelemetryTracer(\"multi-agent-system\")\n        self.metrics = PrometheusMetrics(\"agent_system\")\n\n    async def record_agent_call(self, call: AgentCall) -> None:\n        with self.tracer.start_span(\"agent.invocation\", trace_id=call.trace_id) as span:\n            span.set_attribute(\"agent.id\", call.agent_id)\n            span.set_attribute(\"agent.model\", call.model)\n            span.set_attribute(\"tokens.input\", call.input_tokens)\n            span.set_attribute(\"tokens.output\", call.output_tokens)\n            span.set_attribute(\"latency_ms\", call.latency_ms)\n            span.set_attribute(\"confidence\", call.result.confidence)\n            span.set_attribute(\"cost_cents\", call.estimated_cost_cents)\n\n            if call.error:\n                span.record_exception(call.error)\n                self.metrics.increment(\"agent.errors\", labels={\"agent\": call.agent_id})\n            else:\n                self.metrics.histogram(\"agent.latency\", call.latency_ms,\n                                       labels={\"agent\": call.agent_id})\n```\n\nAgents must never share mutable state directly. Each agent operates on an immutable snapshot of the context it needs, and any state changes are communicated through explicit messages. This prevents race conditions, makes debugging tractable, and enables replay of agent interactions for quality assurance.\n\nBased on post-mortems from both Hermes and LobeHub deployments, these are the failure modes that appear most frequently:\n\nThe temptation is to spawn an agent for every subtask, every validation step, every edge case. This creates exponential cost and latency. Both systems enforce **fan-out caps** at the orchestrator level:\n\n```\ninterface OrchestrationPolicy {\n  maxDepth: number;           // Max nesting of agent calls\n  maxFanOut: number;          // Max concurrent agents\n  maxTotalAgents: number;     // Hard cap per request\n  budgetCapTokens: number;    // Token budget\n  budgetCapUSD: number;       // Cost cap\n  circuitBreaker: {\n    errorRateThreshold: number;  // Trigger if >X% of agents fail\n    timeoutThreshold: number;     // Trigger if avg latency >Y ms\n  };\n}\n```\n\nWhen agents communicate freely, a malicious user input can propagate through the agent network. If Agent A generates a message to Agent B, and that message contains instructions that Agent B treats as authoritative, you have a prompt injection attack vector. Both systems implement:\n\nLLM outputs are probabilistic. Two identical requests to the same agent can produce different results. This makes testing nearly impossible and debugging frustrating. Mitigation strategies:\n\nAs agents accumulate conversation history, the context window fills up quickly. Both systems use aggressive context compression:\n\nNeither Hermes nor LobeHub is universally better. The right architecture depends on your requirements:\n\n| Criterion | Hermes (Top-Down) | LobeHub (Peer-to-Peer) |\n|---|---|---|\nLatency sensitivity |\nBetter—sequential specialists are fast | Worse—consensus rounds add latency |\nCost predictability |\nBetter—bounded execution plans | Worse—dynamic agent counts |\nSolution quality |\nGood for well-defined problems | Better for creative/open-ended problems |\nDebuggability |\nExcellent—linear traces | Poor—emergent behavior |\nScalability |\nLinear with worker pool size | Super-linear with participant count |\nResilience |\nOrchestrator is a bottleneck | True fault tolerance |\nBest for |\nTransactional workflows (support, analytics, coding) | Exploratory workflows (research, design, strategy) |\n\nThe most promising systems in 2025 combine both approaches. They use Hermes-style orchestration for the high-level task decomposition and LobeHub-style consensus for the reasoning-critical subtasks. The result is a system that is both fast and creative, bounded and exploratory.\n\nA typical hybrid flow:\n\nThis hybrid approach is becoming the default pattern for production multi-agent systems because it captures the strengths of both paradigms while mitigating their weaknesses.\n\nIf you're planning to build a multi-agent system in 2025, here's what the evidence from Hermes, LobeHub, and related systems suggests you need:\n\n**Q: Do I really need a custom multi-agent framework, or can I use LangGraph or AutoGen?**\n\nFor simple workflows, yes—those frameworks are fine. But they abstract away the scaling concerns that matter in production: cost control, observability, prompt injection defense, and resilience under load. Hermes and LobeHub's approaches show that these concerns require architectural decisions that generic frameworks don't address. If you're building something that needs to run reliably at scale, investing in purpose-built infrastructure pays off.\n\n**Q: How do I choose between sequential orchestration and peer-to-peer consensus?**\n\nThe rule of thumb: use sequential orchestration when the problem has clear sub-components that can be decomposed and solved independently (data processing, code generation, support ticket resolution). Use consensus-based approaches when the problem requires creative synthesis, trade-off analysis, or when multiple valid answers exist and you need to find the best one (strategy planning, research synthesis, design review). Hybrid approaches get you the best of both worlds.\n\n**Q: What's the realistic cost per complex multi-agent request?**\n\nIn production systems observed in 2025, a well-optimized multi-agent request with 5–8 agents typically costs $0.50–$3.00 depending on complexity and model choices. Unoptimized systems can easily exceed $10 per request. The key differentiator is whether you're using budget-aware routing and context compression. If your costs are higher, you're likely spawning too many agents or failing to compress shared context effectively.\n\nThe 2025 AI agent explosion isn't about making individual agents smarter—it's about building systems where many specialized agents coordinate effectively. Hermes and LobeHub represent two proven points in that design space, and the hybrid architectures they're converging toward suggest the field is still evolving rapidly. The engineers who internalize these patterns now will have a significant advantage as multi-agent systems move from experimental to essential infrastructure.", "url": "https://wpnews.pro/news/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and", "canonical_source": "https://dev.to/tamizuddin/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and-the-2025-ai-42gh", "published_at": "2026-08-16 06:01:34+00:00", "updated_at": "2026-08-16 06:11:17.188881+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-infrastructure", "ai-safety", "developer-tools"], "entities": ["Hermes", "LobeHub", "tamiz.pro"], "alternates": {"html": "https://wpnews.pro/news/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and", "markdown": "https://wpnews.pro/news/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and.md", "text": "https://wpnews.pro/news/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and.txt", "jsonld": "https://wpnews.pro/news/building-multi-agent-systems-that-actually-scale-lessons-from-hermes-lobehub-and.jsonld"}}