Tracing Multi-Agent LLMs: My experience with otel-swarm

A developer using the otel-swarm library to trace multi-agent LLM deployments reports that overwriting the gen_ai.request.model attribute during fallback promotions corrupted analytics, attributing primary model failures to fallback models. The fix records fallback switches as span events instead, preserving accurate p95 latency attribution across 240 model calls and 8 models in a code-gen tool.

Tracing Multi-Agent LLMs: My experience with otel-swarm Hand-wiring OpenTelemetry into every single provider call is tedious, so I've been using a library called otel-swarm to handle the instrumentation. It basically wraps the agent logic so you don't have to manually manage spans for every retry path. The implementation is pretty straightforward. You use task for the root span and agent for the specific roles. js import { createSwarm } from 'otel-swarm'; const swarm = createSwarm { service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' } ; await swarm.task 'generation', { 'my.prompt': prompt }, async root = { const plan = await swarm.agent 'planner', = swarm.llm 'planner', { model: 'primary-model-id', fallbackModel: 'fallback-model-id', call: async model = { const res = await yourProviderCall model ; return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out }; } } ; await swarm.agent 'critic', async span = { swarm.reviewEvents span, issues ; } ; } ; In my real-world deployment with a code-gen tool, this setup tracked over 240 model calls across 8 different models. The key is that swarm.events mirrors the span lifecycle, meaning the traceId allows me to deep-link from a UI row directly to the exact trace in SigNoz. One huge technical lesson I learned the hard way: never put fallback model info in the attribute you use for grouping. Initially, I had the system overwrite the gen ai.request.model attribute when a fallback kicked in. This sounded logical—the span should "tell the truth" about who answered. But it wrecked my analytics. Every dashboard panel grouping by model attributed the primary model's timeout and wasted latency to the fallback model. My expensive primary looked flawless, while the cheap fallback looked like it had terrible p95 latency. Now, I record the switch as a fallback promotion span event. It keeps the data clean and ensures the primary model gets the "blame" for the failure it actually caused. This is a much better AI workflow for anyone doing a deep dive into LLM observability. Next Mid-Chain Governance: Fixing the AI Agent Blind Spot → /en/threads/3806/