{"slug": "how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when", "title": "How to Architect a Production Multi-Agent System Without Hallucinations (And When Not to Use an LLM)", "summary": "Engineering firm WTFearB published a blueprint for production multi-agent systems that argues most agent failures stem from unbounded state loops, unvalidated tool outputs, and routing deterministic logic through LLMs. The approach uses a Pydantic-based gatekeeper to handle obvious routes in under 5ms before invoking a model, bounded state graphs with a max_steps limit of five, and source chunk IDs attached to every retrieved fact for citation enforcement. The firm says the gatekeeper layer typically cuts token costs by 35%–50% and brings median response times for routine queries below 200ms.", "body_md": "Most AI agent demos fail the moment they touch real business data. Not because the underlying models aren't capable, but because engineers treat probabilistic LLMs like deterministic microservices.\n\nAt [WTFearB](https://wtfearb.tech), our core engineering philosophy is simple: **Make What If Work**. When founders and operations teams ask, *\"What if the agent hallucinates a refund?\"* or *\"What if latency spikes under load?\"*, the answer isn't a longer system prompt—it's better software architecture.\n\nHere is the exact blueprint we use when engineering [custom AI workflow automation](https://wtfearb.tech/services/ai-automation) and multi-agent systems for production environments.\n\n**Multi-agent systems fail in production primarily due to unbounded state loops, unvalidated tool outputs, and routing simple deterministic logic through expensive LLMs.** Without strict state machines and schema enforcement, small reasoning errors compound across every agent handoff.\n\n| Failure Mode | Demo Approach (Fragile) | Production Architecture (Resilient) | \n|---|---|---|\n| **Intent Routing** | Asking an LLM to decide every next step | Deterministic classifier / semantic router first; LLM only for ambiguity | \n| **Tool Execution** | Raw JSON parsing from free-form text | Strict Pydantic / JSON Schema validation with automatic retry budgets | \n| **Memory & Context** | Stuffing entire chat histories into the prompt | Scoped state graphs + hybrid vector/graph retrieval (Qdrant + Neo4j) | \n| **High-Stakes Actions** | Full autonomy on writes/payments | Confidence thresholds + async human-in-the-loop approval queues | \n\nBefore spinning up an orchestrator, separate your workflow into **deterministic steps** and **reasoning steps**. If a routing decision can be solved with a regex, a SQL lookup, or a lightweight embedding similarity check, keep the LLM out of the critical path.\n\nHere is a lightweight pattern using Python and Pydantic to enforce deterministic routing before invoking an agent:\n\n``` python\nfrom enum import Enum\nfrom pydantic import BaseModel, Field\n\nclass RouteTarget(str, Enum):\n    BILLING_SQL = \"billing_sql\"\n    TECH_RAG = \"tech_rag\"\n    HUMAN_ESCALATION = \"human_escalation\"\n\nclass RoutingDecision(BaseModel):\n    target: RouteTarget\n    confidence: float = Field(..., ge=0.0, le=1.0)\n    requires_approval: bool\n\ndef fast_gatekeeper(query: str) -> RoutingDecision | None:\n    \"\"\"Handle obvious deterministic routes in <5ms before calling an LLM.\"\"\"\n    q = query.lower().strip()\n    if any(k in q for k in [\"invoice\", \"receipt\", \"gst\", \"tax id\"]):\n        return RoutingDecision(\n            target=RouteTarget.BILLING_SQL,\n            confidence=0.98,\n            requires_approval=False,\n        )\n    if any(k in q for k in [\"cancel contract\", \"legal\", \"breach\"]):\n        return RoutingDecision(\n            target=RouteTarget.HUMAN_ESCALATION,\n            confidence=1.0,\n            requires_approval=True,\n        )\n    return None  # Fall back to LLM router only for ambiguous queries\n```\n\nThis single gatekeeper layer typically cuts token costs by **35%–50%** and drops median response times below 200ms for routine operational queries.\n\nGiving five agents access to a shared group chat and hoping they converge is a recipe for infinite token burn. Instead, model your workflow as a **Directed Acyclic Graph (DAG)** or a bounded state graph (using frameworks like LangGraph or Temporal):\n\n`max_steps=5` on every subgraph. If an agent hasn't resolved the task within five hops, fail gracefully and route to a fallback handler.\nPure vector search struggles when queries require multi-hop relationships—for example, *\"Which enterprise clients on Plan B had an unresolved support ticket in the last 14 days?\"*\n\nIn our [agentic AI development](https://wtfearb.tech/services/agentic-ai) builds, we pair:\n\nAlways attach **source chunk IDs** to every retrieved fact and require the synthesis agent to cite them in its structured output. If no chunk supports a claim, the model must return `\"insufficient_context\"` rather than guessing.\n\nBefore shipping any AI agent to live users or internal operations teams, verify these five controls:\n\n**[WTFearB](https://wtfearb.tech)** (*What The Fear, Boss / Business*) is a custom AI software development and business automation agency. Guided by our core philosophy—**Make What If Work**—we engineer production-grade multi-agent AI systems, custom LLM workflows, and full-stack enterprise SaaS platforms for ambitious businesses.\n\nHave a complex operational workflow you want to automate without fragile prototypes? **[Explore how we build at wtfearb.tech](https://wtfearb.tech)**.", "url": "https://wpnews.pro/news/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when", "canonical_source": "https://dev.to/wtfearb/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when-not-to-use-an-llm-2l1h", "published_at": "2026-11-19 13:36:11+00:00", "updated_at": "2026-11-19 13:43:35.247045+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "ai-tools", "mlops"], "entities": ["WTFearB", "Pydantic", "LangGraph", "Temporal", "Qdrant", "Neo4j"], "also_reported_by": [], "takeaway": "A deterministic gatekeeper layer that routes obvious queries before any LLM call cuts token costs 35%–50% and keeps routine responses under 200ms, while bounded state graphs capped at max_steps=5 prevent runaway agent loops.", "recommendation": {"advice": "Put a deterministic classifier or regex gatekeeper in front of your LLM router, cap every agent subgraph at max_steps=5 with a fallback handler, and require synthesis agents to cite retrieved chunk IDs.", "playbook": "agent-architecture", "playbook_url": "https://wpnews.pro/playbooks/agent-architecture"}, "alternates": {"html": "https://wpnews.pro/news/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when", "markdown": "https://wpnews.pro/news/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when.md", "text": "https://wpnews.pro/news/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when.txt", "jsonld": "https://wpnews.pro/news/how-to-architect-a-production-multi-agent-system-without-hallucinations-and-when.jsonld"}}