{"slug": "how-i-built-and-broke-a-production-multi-agent-ai-system", "title": "How I Built (and Broke) a Production Multi-Agent AI System", "summary": "A developer built a production-style multi-agent orchestration system using Python/FastAPI, LangGraph, PostgreSQL, Redis, ChromaDB, Celery, OpenTelemetry, and React, then documented three bugs encountered after the initial build. The failures included an escalation logic flaw that generated over 5,000 duplicate review_requests rows for a single task, a silent checkpointer fallback that let the system run without state persistence, and a replay feature that compared OpenTelemetry span_ids against LangGraph checkpoint_ids, causing it to silently reconstruct approximate state instead of loading real checkpoints. The developer fixed each by adding a database-level unique constraint, removing silent fallbacks in favor of hard RuntimeErrors, and recording the actual checkpoint_id as a span attribute, concluding that fallbacks masking real problems are worse than no fallback at all.", "body_md": "Most agent demos work great — right up until you run them past the happy path. This is the story of building a real multi-agent orchestration system, and three bugs that taught me more than the initial build did.\n\nA production-style agent system with:\n\nThe stack: Python/FastAPI, LangGraph, PostgreSQL, Redis, ChromaDB, Celery, OpenTelemetry, React.\n\nThe architecture is the easy part to explain. The bugs are the part worth writing about.\n\nEarly on, I noticed one test task had generated over 5,000 duplicate rows in the `review_requests` table. A single task. One escalation. Five thousand rows.\n\nThe root cause: the human-in-the-loop escalation logic had no protection against a task being re-evaluated multiple times in quick succession — each re-evaluation created a new pending review request instead of checking if one already existed. Under certain retry/timing conditions, this spiraled.\n\nThe fix had two layers:\n\n`UNIQUE (task_id, subtask_id) WHERE status = 'pending'`\nThe app-level check alone wasn't enough — there was still a race condition window between the check and the insert. The database constraint is what actually closed the gap. Lesson: if a bug can be prevented at the database layer, prevent it there. Application logic is not atomic by default.\n\nWhile building the observability layer, I had a checkpointer initialization path with a fallback: if the primary checkpointer setup failed, the code would fall back to a sentinel value and continue running — silently, without persistence.\n\nThis is worse than a crash. A crash tells you something is wrong. A silent fallback lets the whole system appear to work while quietly not doing the one thing (state persistence) the rest of the architecture depended on.\n\nThe fix: removed the fallback entirely. If checkpointer init fails now, it raises a hard `RuntimeError` at startup, with a clear log line confirming success when it *does* work. No degraded-but-invisible states allowed.\n\nThe replay feature let you re-run a past task with a modified input, comparing the old and new outcomes side by side. It worked — until I actually verified *what* it was comparing.\n\nThe frontend was sending an OpenTelemetry `span_id` to the replay endpoint, where a LangGraph `checkpoint_id` was expected. These are two unrelated identifier systems that happen to look like similar opaque strings. The backend's `load_checkpoint()` function had its own fallback (see a pattern here?) that would silently reconstruct approximate state from trace data instead of loading the real checkpoint when the ID didn't resolve.\n\nSo replay was \"working\" — it just wasn't replaying from the real checkpoint most of the time.\n\nFix: recorded the actual `checkpoint_id` as a span attribute at trace time, and had the frontend send *that* instead of the span's own id. I also removed the silent fallback in `load_checkpoint()` — now it raises an explicit error if the checkpoint can't be found, rather than quietly approximating. I verified the fix by checking for a `[PATH: NATIVE_LANGGRAPH_CHECKPOINT]` log line confirming the real path was actually being used.\n\nEvery one of these bugs shares something: **a system that appeared to work while silently doing the wrong thing.** Not crashes — those are easy to catch. Silent degradation is the dangerous failure mode, because your test suite can pass, your demo can look perfect, and the bug only surfaces under conditions you didn't think to simulate.\n\nIf I had to generalize this into a rule for building agent systems: **wherever your code has a fallback \"just in case,\" ask whether that fallback should actually be a hard failure instead.** A fallback that masks a real problem is worse than no fallback at all.\n\nI packaged the system (with all three fixes already in place) into a starter kit for developers building similar agent products, since most of the pain in this space isn't the \"agent\" logic — it's this orchestration/memory/observability layer that everyone ends up rebuilding. If you're curious: [https://whop.com/gauravxd/multi-agent-orchestration-kit/](https://whop.com/gauravxd/multi-agent-orchestration-kit/)\n\nBut even if you never look at the kit, I hope the bugs are useful on their own. If you're building agent systems and want to compare notes on failure modes you've hit, I'd genuinely like to hear about them.", "url": "https://wpnews.pro/news/how-i-built-and-broke-a-production-multi-agent-ai-system", "canonical_source": "https://dev.to/gauravstack/how-i-built-and-broke-a-production-multi-agent-ai-system-4b2i", "published_at": "2026-09-21 10:02:50+00:00", "updated_at": "2026-09-21 10:31:29.772018+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "mlops"], "entities": ["LangGraph", "FastAPI", "PostgreSQL", "Redis", "ChromaDB", "Celery", "OpenTelemetry", "React"], "alternates": {"html": "https://wpnews.pro/news/how-i-built-and-broke-a-production-multi-agent-ai-system", "markdown": "https://wpnews.pro/news/how-i-built-and-broke-a-production-multi-agent-ai-system.md", "text": "https://wpnews.pro/news/how-i-built-and-broke-a-production-multi-agent-ai-system.txt", "jsonld": "https://wpnews.pro/news/how-i-built-and-broke-a-production-multi-agent-ai-system.jsonld"}}