Most agent demos work great — right up until you run them past the happy path. This is the story of building a real multi-agent orchestration system, and three bugs that taught me more than the initial build did.
A production-style agent system with: The stack: Python/FastAPI, LangGraph, PostgreSQL, Redis, ChromaDB, Celery, OpenTelemetry, React.
The architecture is the easy part to explain. The bugs are the part worth writing about.
Early on, I noticed one test task had generated over 5,000 duplicate rows in the review_requests table. A single task. One escalation. Five thousand rows.
The root cause: the human-in-the-loop escalation logic had no protection against a task being re-evaluated multiple times in quick succession — each re-evaluation created a new pending review request instead of checking if one already existed. Under certain retry/timing conditions, this spiraled.
The fix had two layers:
UNIQUE (task_id, subtask_id) WHERE status = 'pending'
The app-level check alone wasn't enough — there was still a race condition window between the check and the insert. The database constraint is what actually closed the gap. Lesson: if a bug can be prevented at the database layer, prevent it there. Application logic is not atomic by default.
While building the observability layer, I had a checkpointer initialization path with a fallback: if the primary checkpointer setup failed, the code would fall back to a sentinel value and continue running — silently, without persistence. This is worse than a crash. A crash tells you something is wrong. A silent fallback lets the whole system appear to work while quietly not doing the one thing (state persistence) the rest of the architecture depended on.
The fix: removed the fallback entirely. If checkpointer init fails now, it raises a hard RuntimeError at startup, with a clear log line confirming success when it does work. No degraded-but-invisible states allowed.
The replay feature let you re-run a past task with a modified input, comparing the old and new outcomes side by side. It worked — until I actually verified what it was comparing.
The frontend was sending an OpenTelemetry span_id to the replay endpoint, where a LangGraph checkpoint_id was expected. These are two unrelated identifier systems that happen to look like similar opaque strings. The backend's load_checkpoint() function had its own fallback (see a pattern here?) that would silently reconstruct approximate state from trace data instead of the real checkpoint when the ID didn't resolve.
So replay was "working" — it just wasn't replaying from the real checkpoint most of the time.
Fix: recorded the actual checkpoint_id as a span attribute at trace time, and had the frontend send that instead of the span's own id. I also removed the silent fallback in load_checkpoint() — now it raises an explicit error if the checkpoint can't be found, rather than quietly approximating. I verified the fix by checking for a [PATH: NATIVE_LANGGRAPH_CHECKPOINT] log line confirming the real path was actually being used.
Every one of these bugs shares something: a system that appeared to work while silently doing the wrong thing. Not crashes — those are easy to catch. Silent degradation is the dangerous failure mode, because your test suite can pass, your demo can look perfect, and the bug only surfaces under conditions you didn't think to simulate.
If I had to generalize this into a rule for building agent systems: wherever your code has a fallback "just in case," ask whether that fallback should actually be a hard failure instead. A fallback that masks a real problem is worse than no fallback at all. I packaged the system (with all three fixes already in place) into a starter kit for developers building similar agent products, since most of the pain in this space isn't the "agent" logic — it's this orchestration/memory/observability layer that everyone ends up rebuilding. If you're curious: https://whop.com/gauravxd/multi-agent-orchestration-kit/
But even if you never look at the kit, I hope the bugs are useful on their own. If you're building agent systems and want to compare notes on failure modes you've hit, I'd genuinely like to hear about them.