{"slug": "i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me", "title": "I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me.", "summary": "A developer self-hosted SigNoz to trace a multi-step LLM agent pipeline, finding the GenAI semantic-convention attributes the most useful feature. The setup required only a minimal Foundry CLI deployment and stock OpenTelemetry SDK, with no vendor-specific code. The trace detail view provided clear visibility into each agent step and LLM call, including model, token usage, and cost.", "body_md": "Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline \"works\", the answer is just bad, slow or three times more expensive than yesterday. `print()`\n\ndebugging tells you nothing, because the interesting question is never \"did step 3 run\". It is \"what did step 3 see, which model actually answered and what did it cost\".\n\nSo I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant (plan, retrieve, generate, synthesize) instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes (`gen_ai.request.model`\n\n, `gen_ai.usage.input_tokens`\n\nand friends) on every LLM call.\n\nThis post is about the feature that turned out to be the most useful. It was not the one I expected.\n\nSelf-hosting used to mean wrangling a long docker-compose file. SigNoz now ships **Foundry**, a small CLI that casts the whole stack:\n\n```\ncurl -fsSL https://signoz.io/foundry.sh | bash   # installs foundryctl (checksum-verified)\nfoundryctl cast -f casting.yaml                  # deploys the full stack on Docker\n```\n\nwith a `casting.yaml`\n\nthat is all of eight lines:\n\n```\napiVersion: v1alpha1\nkind: Installation\nmetadata:\n  name: signoz\nspec:\n  deployment:\n    flavor: compose\n    mode: docker\n```\n\nA few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on `localhost:8080`\n\nand the collector listens on `4317`\n\n(gRPC) and `4318`\n\n(HTTP).\n\n**One gotcha worth knowing:** telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers \"cannot create agent without orgId\" and the OTLP ports reset every connection. If your exporter logs `Connection reset by peer`\n\non a fresh install, you haven't finished the two-minute signup at `localhost:8080`\n\nyet. Create the account and ingestion starts working within about thirty seconds, no restarts needed.\n\nMy demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to `localhost:4318`\n\n. That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists.\n\nEach simulated request produces one trace:\n\n```\nagent.request                    (root, 2.02s)\n├── agent.plan                   (850ms)\n│   └── gen_ai.generate plan     (696ms, gen_ai.* attributes)\n├── agent.retrieve               (461ms, sometimes errors: vector store timeout)\n├── agent.generate               (564ms)\n│   └── gen_ai.generate answer   (564ms, gen_ai.* attributes)\n└── agent.synthesize             (139ms, sometimes errors: citation validation)\n```\n\nEvery `gen_ai.generate`\n\nspan carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage (`gen_ai.client.token.usage`\n\n) and estimated spend (`agent.llm.cost`\n\n), both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically.\n\nSixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up `research-assistant`\n\non its own with RED metrics already computed (p99 latency, error rate, throughput). I wrote no configuration for that.\n\nAnd it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any `gen_ai.generate`\n\nspan opens a details panel with every attribute I set: model `llama-4-maverick`\n\n, provider `meta`\n\n, 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. \"Why did this request take four seconds\" stops being a mystery and becomes a picture.\n\nBut a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it.\n\nHere is the moment it clicked. In the Trace Explorer I typed:\n\n```\ngen_ai.request.model = 'qwen3-coder-plus'\n```\n\nTwo things happened. First, the autocomplete suggested `qwen3-coder-plus`\n\nbefore I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system.\n\nStop and consider what that means for agent debugging. I never told SigNoz what `gen_ai.request.model`\n\nis. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary *becomes the product's vocabulary*. For LLM systems, where all the interesting facts live in custom attributes (model, provider, token counts, agent role, tool name), this is the difference between an observability tool that fits and one you fight with.\n\nAnd the same query keeps working as you move across signals:\n\n`trace_id = '<this trace>'`\n\npre-filled and the time window auto-scoped. I got back exactly the two log lines belonging to that request, correlated purely by the trace context the OTel logging handler injects. Nobody parses log lines to find a request id. The correlation is structural.`gen_ai.client.token.usage`\n\nshowing 6 time series (3 models times 2 token types), `agent.llm.cost`\n\nin usd showing 3. Cost per model is a group-by away, using the exact attribute names from my instrumentation.Most observability stacks treat custom attributes as second-class blobs that need schema work before they are queryable. SigNoz treats them as the whole point. For agent systems, they are.\n\nGoing deep on one feature meant walking past a lot of others. Quick notes from the tour, agent-flavored:\n\nEach of these uses the attribute vocabulary your instrumentation defines. That is the theme of the whole product, and it is why the one feature I picked is really the foundation the rest stand on.\n\nThe uncomfortable truth about multi-agent LLM systems is that their most important behavior (which model ran, what it consumed, what it cost, why it was retried) is invisible to conventional monitoring. It lives entirely in domain-specific span attributes. A backend that makes those attributes instantly queryable, correlatable across traces and logs, and promotable into dashboards and alerts is not a nice-to-have there. It is the debugger.\n\nThe whole experiment cost me an afternoon: one CLI install, eight lines of YAML, a stock OTel SDK and zero vendor code in the app. I'm taking this setup into the Agents of SigNoz hackathon (July 20 to 26), where the plan is considerably less simulated. If you're building anything agent-shaped, self-host SigNoz and type one of your own attribute names into the Trace Explorer. That autocomplete dropdown is the moment you'll get it too.\n\n*The demo app is a 170-line Python script using only opentelemetry-sdk and the OTLP HTTP exporter. Stack: SigNoz self-hosted via Foundry on Docker (ClickHouse + Postgres + OTel collector), macOS host.*\n\n*Demo code, self-host config and blog source: github.com/himanshu748/signoz-agent-observability. Team 404Found.*", "url": "https://wpnews.pro/news/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me", "canonical_source": "https://dev.to/himanshu_748/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me-4k71", "published_at": "2026-07-11 08:22:06+00:00", "updated_at": "2026-07-11 08:43:28.954602+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "developer-tools", "ai-infrastructure", "mlops"], "entities": ["SigNoz", "Foundry", "OpenTelemetry", "ClickHouse", "Postgres", "llama-4-maverick", "Meta"], "alternates": {"html": "https://wpnews.pro/news/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me", "markdown": "https://wpnews.pro/news/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me.md", "text": "https://wpnews.pro/news/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me.txt", "jsonld": "https://wpnews.pro/news/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me.jsonld"}}