{"slug": "tracing-multi-agent-llms-my-experience-with-otel-swarm", "title": "Tracing Multi-Agent LLMs: My experience with otel-swarm", "summary": "A developer using the otel-swarm library to trace multi-agent LLM deployments reports that overwriting the gen_ai.request.model attribute during fallback promotions corrupted analytics, attributing primary model failures to fallback models. The fix records fallback switches as span events instead, preserving accurate p95 latency attribution across 240 model calls and 8 models in a code-gen tool.", "body_md": "# Tracing Multi-Agent LLMs: My experience with otel-swarm\n\nHand-wiring OpenTelemetry into every single provider call is tedious, so I've been using a library called otel-swarm to handle the instrumentation. It basically wraps the agent logic so you don't have to manually manage spans for every retry path.\n\nThe implementation is pretty straightforward. You use `task()`\n\nfor the root span and `agent()`\n\nfor the specific roles.\n\n``` js\nimport { createSwarm } from 'otel-swarm';\n\nconst swarm = createSwarm({ service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' });\n\nawait swarm.task('generation', { 'my.prompt': prompt }, async (root) => {\n const plan = await swarm.agent('planner', () =>\n swarm.llm('planner', {\n model: 'primary-model-id',\n fallbackModel: 'fallback-model-id',\n call: async (model) => {\n const res = await yourProviderCall(model);\n return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out };\n }\n })\n );\n\n await swarm.agent('critic', async (span) => {\n swarm.reviewEvents(span, issues); \n });\n});\n```\n\nIn my real-world deployment with a code-gen tool, this setup tracked over 240 model calls across 8 different models. The key is that `swarm.events`\n\nmirrors the span lifecycle, meaning the `traceId`\n\nallows me to deep-link from a UI row directly to the exact trace in SigNoz.\n\nOne huge technical lesson I learned the hard way: never put fallback model info in the attribute you use for grouping.\n\nInitially, I had the system overwrite the `gen_ai.request.model`\n\nattribute when a fallback kicked in. This sounded logical—the span should \"tell the truth\" about who answered. But it wrecked my analytics. Every dashboard panel grouping by model attributed the primary model's timeout and wasted latency to the fallback model. My expensive primary looked flawless, while the cheap fallback looked like it had terrible p95 latency.\n\nNow, I record the switch as a `fallback_promotion`\n\nspan event. It keeps the data clean and ensures the primary model gets the \"blame\" for the failure it actually caused. This is a much better AI workflow for anyone doing a deep dive into LLM observability.\n\n[Next Mid-Chain Governance: Fixing the AI Agent Blind Spot →](/en/threads/3806/)", "url": "https://wpnews.pro/news/tracing-multi-agent-llms-my-experience-with-otel-swarm", "canonical_source": "https://promptcube3.com/en/threads/3831/", "published_at": "2026-07-26 20:02:01+00:00", "updated_at": "2026-07-26 20:08:48.060908+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["otel-swarm", "SigNoz"], "alternates": {"html": "https://wpnews.pro/news/tracing-multi-agent-llms-my-experience-with-otel-swarm", "markdown": "https://wpnews.pro/news/tracing-multi-agent-llms-my-experience-with-otel-swarm.md", "text": "https://wpnews.pro/news/tracing-multi-agent-llms-my-experience-with-otel-swarm.txt", "jsonld": "https://wpnews.pro/news/tracing-multi-agent-llms-my-experience-with-otel-swarm.jsonld"}}