Sarvar's post about discovering a hidden retry in a 5-agent pipeline (one agent taking 22.6s while others took 5s) is a perfect case study in why observability infrastructure matters for agentic systems.
Here's what jumped out:
Agent-as-black-box is dangerous. When you string together multiple agents, you lose visibility into retry logic, backoff strategies, and cascade failures unless you instrument at the span level. The latency wasn't in the agent logic itself; it was in the retry envelope.
Span hierarchy exposes the invisible. Sentry's approach of grouping spans hierarchically made the problem visible at a glance. Without it, you'd see "agent took 22.6s" and assume it was compute-bound. With hierarchy, the retry pattern was obvious.
This scales badly across agents. In a 5-agent system, one bad retry strategy can block or cascade. Add error handling, timeout logic, and fallback chains, and you're building a retry forest no one fully understands. The observability debt compounds.
The fix is cheap, the insight is priceless. Once Sarvar knew what was happening, tuning retry counts or backoff curves took minutes. The time cost was finding it.
Takeaway: If you're building multi-agent systems, instrument early. Span-level observability isn't optional; it's the difference between "it's slow" and "here's why, and here's the fix."