{"slug": "tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack", "title": "Tracing a multi-agent LLM system: otel-swarm and a SigNoz dashboard pack", "summary": "A developer extracted OpenTelemetry instrumentation from DevSwarm into an MIT library called otel-swarm, which traces multi-agent LLM systems with a single createSwarm() call and three verbs: task(), agent(), and llm(). The library records fallback promotions and critic catches as span events, enabling full trace trees in SigNoz dashboards. In production, it has traced 29 generations, 243 model calls, and 3.78 million tokens.", "body_md": "A single LLM call is easy to reason about. You send a prompt, you get tokens back, you log the latency and you move on.\n\nA swarm is not that. Four or five agents run, some in parallel, each with its own model. One of them times out and quietly falls back to a cheaper model. A critic agent reads the output and sends work back for another round. When the whole thing takes 40 seconds instead of 12, or costs three times what you budgeted, you have no idea which agent did it. Your logs are a flat stream of interleaved lines from concurrent tasks, with no parent-child structure and no way to ask \"which role burned the tokens\".\n\nTraces solve this exactly. The catch is that hand-wiring OpenTelemetry across every role, every provider call and every retry path is boring work that nobody wants to do twice, and if you skip a level the trace tree lies to you.\n\nSo I extracted the instrumentation out of DevSwarm (a multi-agent code generator I built for this hackathon) into a standalone MIT library: [otel-swarm](https://github.com/himanshu748/otel-swarm).\n\nOne `createSwarm()`\n\ncall, then three verbs.\n\n``` js\nimport { createSwarm } from 'otel-swarm';\n\nconst swarm = createSwarm({ service: 'my-swarm', otlpEndpoint: 'http://localhost:4318' });\n\nawait swarm.task('generation', { 'my.prompt': prompt }, async (root) => {\n  const plan = await swarm.agent('planner', () =>\n    swarm.llm('planner', {\n      model: 'primary-model-id',\n      fallbackModel: 'fallback-model-id',\n      call: async (model) => {\n        const res = await yourProviderCall(model);\n        return { content: res.text, inputTokens: res.usage.in, outputTokens: res.usage.out };\n      }\n    })\n  );\n\n  await swarm.agent('critic', async (span) => {\n    swarm.reviewEvents(span, issues);   // each issue becomes a critic_catch span event\n  });\n});\n\nswarm.events.on('event', (e) => { /* llm_start, llm_end, fallback, critic_catch */ });\n```\n\n`task()`\n\nopens the root span, `agent()`\n\nopens a child span per role and `llm()`\n\nopens a GenAI-semconv span per model call. `call(model)`\n\nis invoked again with `fallbackModel`\n\nif the primary throws, and the switch is recorded as a `fallback_promotion`\n\nspan event carrying `from`\n\n, `to`\n\nand the verbatim provider error.\n\nThe last line matters more than it looks. `swarm.events`\n\nis an EventEmitter that mirrors the span lifecycle, and LLM events carry the `traceId`\n\n. A live UI reads the emitter, the tracing backend reads the OTLP exporter and both are fed from the same code path, so the dashboard and the UI can never disagree about what happened. The `traceId`\n\nmeans a row in your UI deep-links to the exact trace in SigNoz.\n\nRunning `npm run example`\n\nagainst a local SigNoz produced 9 spans under service `otel-swarm-demo`\n\n: a root `generation`\n\nspan of 1325ms, then `agent.planner`\n\n/`llm.planner`\n\n, `agent.frontend`\n\n/`llm.frontend`\n\n, `agent.backend`\n\n/`llm.backend`\n\nand `agent.critic`\n\n/`llm.critic`\n\n. `llm.frontend`\n\ncarried a `fallback_promotion`\n\nevent and `agent.critic`\n\ncarried a `critic_catch`\n\nevent. The whole story is one trace.\n\nIn production use inside DevSwarm the same library has traced 29 generations, 243 model calls across 8 models, 3.78 million tokens and 257 review catches.\n\nThis one cost me an afternoon.\n\nAn earlier version of the library did the obvious thing when a primary model failed: it overwrote `gen_ai.request.model`\n\nwith the fallback's name, so the span would \"tell the truth\" about which model actually answered.\n\nThat is wrong, and it is wrong in a way that is hard to see. Every dashboard panel that groups by model then attributes the primary's failure, its timeout and all of its wasted latency to the fallback that cleaned up after it. My \"tokens and latency by model\" table showed a cheap fallback model with terrible p95 latency and my expensive primary looking flawless, because every time the primary blew up its cost was silently reassigned to whoever picked up the pieces. I spent an afternoon convinced the wrong model was slow.\n\nThe rule that comes out of it: **the attribute you group by must record the model you attempted, not the model that ended up answering.** The promotion is a discrete thing that happened during the span, and a discrete thing that happened during a span is what span events are for:\n\n```\nspan.addEvent('fallback_promotion', { from: model, to: fallbackModel, reason: String(err.message || err) });\n```\n\nNow the grouped panel keeps attributing failure to whoever caused it, and \"how often does this role get promoted\" is a separate query over events. Two different questions, two different storage locations, no cross-contamination.\n\nThe general form: if a value can change mid-span, it does not belong on an attribute you aggregate over.\n\nThe consequence of putting things in events is that you now have to query events, and SigNoz stores them as an array of JSON strings on the span row. There is no autocomplete that will lead you here. The pattern is `ARRAY JOIN`\n\nto flatten the array into one row per event, then `JSONExtractString`\n\nto reach into the event's `attributeMap`\n\n:\n\n```\nSELECT JSONExtractString(e, 'attributeMap', 'severity') AS severity,\n       JSONExtractString(e, 'attributeMap', 'target') AS target,\n       count() AS catches\nFROM signoz_traces.distributed_signoz_index_v3\nARRAY JOIN events AS e\nWHERE serviceName = '{{.service}}' AND name = 'agent.critic'\n  AND JSONExtractString(e, 'name') = 'critic_catch'\nGROUP BY severity, target ORDER BY catches DESC\n```\n\nThat gives you review findings broken down by severity and by what they were aimed at, straight out of span events, with no separate metrics pipeline.\n\nWhen you only need to know whether an event fired at all, do not pay for the `ARRAY JOIN`\n\n. `arrayExists`\n\nwith a substring test over the raw array is enough and it keeps one row per span, which is what you want for a time series:\n\n```\nSELECT toStartOfInterval(timestamp, INTERVAL 15 MINUTE) AS ts,\n       attributes_string['swarm.role'] AS role,\n       count() AS value\nFROM signoz_traces.distributed_signoz_index_v3\nWHERE serviceName = '{{.service}}'\n  AND arrayExists(e -> e LIKE '%fallback_promotion%', events)\n  AND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}\nGROUP BY ts, role ORDER BY ts\n```\n\nThe repo ships 3 importable SigNoz dashboards (POST each JSON to `/api/v1/dashboards`\n\n). Every query uses a `{{.service}}`\n\ndashboard variable that defaults to `otel-swarm-demo`\n\n, so you point the pack at your own service by editing one dropdown instead of doing find-and-replace across query strings.\n\n`arrayExists`\n\nquery above).Two rules ship in the v2alpha1 schema (POST to `/api/v2/rules`\n\n).\n\nThe first is a fallback-promotion spike. If promotions jump, a provider is degrading and you are silently paying a different bill than you planned.\n\nThe second is the one I care about more: a **review catch-rate flatline**. If your critic agent suddenly stops finding anything, the tempting read is that the generators got better. In practice the reviewer broke: a prompt change made it answer in a shape the parser drops, or its model started returning empty content and the failure is being swallowed as \"no issues found\". A quality gate that passes everything is indistinguishable from no quality gate, and it fails silently by construction. Alert on the absence.\n\nThe repo also includes `casting.yaml`\n\nand `casting.yaml.lock`\n\n, the Foundry config the SigNoz instance behind these dashboards was installed from, so the backend the pack targets is reproducible rather than assumed.\n\n```\nnpm install github:himanshu748/otel-swarm\n```\n\nTo see the example trace end to end against a local SigNoz:\n\n```\ngit clone https://github.com/himanshu748/otel-swarm && cd otel-swarm\nnpm install\nnpm run example                                                     # spans to console\nOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 npm run example   # spans to SigNoz\n```\n\nWith no endpoint set it exports to the console, so you can check the span tree before wiring a backend.\n\nThe library is about 130 lines. Most of the value is not code, it is the two decisions above: keep mutable facts in span events, and treat \"the reviewer found nothing\" as a symptom rather than a result. Both were learned by getting them wrong first.\n\nI built otel-swarm with Claude Code, including the dashboard JSON and the ClickHouse queries in this post.\n\nRepo: [github.com/himanshu748/otel-swarm](https://github.com/himanshu748/otel-swarm), MIT.", "url": "https://wpnews.pro/news/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack", "canonical_source": "https://dev.to/himanshu_748/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack-4m85", "published_at": "2026-07-26 17:40:05+00:00", "updated_at": "2026-07-26 18:00:20.889479+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["otel-swarm", "DevSwarm", "SigNoz", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack", "markdown": "https://wpnews.pro/news/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack.md", "text": "https://wpnews.pro/news/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack.txt", "jsonld": "https://wpnews.pro/news/tracing-a-multi-agent-llm-system-otel-swarm-and-a-signoz-dashboard-pack.jsonld"}}