I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me. A developer self-hosted SigNoz to trace a multi-step LLM agent pipeline, finding the GenAI semantic-convention attributes the most useful feature. The setup required only a minimal Foundry CLI deployment and stock OpenTelemetry SDK, with no vendor-specific code. The trace detail view provided clear visibility into each agent step and LLM call, including model, token usage, and cost. Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline "works", the answer is just bad, slow or three times more expensive than yesterday. print debugging tells you nothing, because the interesting question is never "did step 3 run". It is "what did step 3 see, which model actually answered and what did it cost". So I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant plan, retrieve, generate, synthesize instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes gen ai.request.model , gen ai.usage.input tokens and friends on every LLM call. This post is about the feature that turned out to be the most useful. It was not the one I expected. Self-hosting used to mean wrangling a long docker-compose file. SigNoz now ships Foundry , a small CLI that casts the whole stack: curl -fsSL https://signoz.io/foundry.sh | bash installs foundryctl checksum-verified foundryctl cast -f casting.yaml deploys the full stack on Docker with a casting.yaml that is all of eight lines: apiVersion: v1alpha1 kind: Installation metadata: name: signoz spec: deployment: flavor: compose mode: docker A few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on localhost:8080 and the collector listens on 4317 gRPC and 4318 HTTP . One gotcha worth knowing: telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers "cannot create agent without orgId" and the OTLP ports reset every connection. If your exporter logs Connection reset by peer on a fresh install, you haven't finished the two-minute signup at localhost:8080 yet. Create the account and ingestion starts working within about thirty seconds, no restarts needed. My demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to localhost:4318 . That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists. Each simulated request produces one trace: agent.request root, 2.02s ├── agent.plan 850ms │ └── gen ai.generate plan 696ms, gen ai. attributes ├── agent.retrieve 461ms, sometimes errors: vector store timeout ├── agent.generate 564ms │ └── gen ai.generate answer 564ms, gen ai. attributes └── agent.synthesize 139ms, sometimes errors: citation validation Every gen ai.generate span carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage gen ai.client.token.usage and estimated spend agent.llm.cost , both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically. Sixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up research-assistant on its own with RED metrics already computed p99 latency, error rate, throughput . I wrote no configuration for that. And it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any gen ai.generate span opens a details panel with every attribute I set: model llama-4-maverick , provider meta , 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. "Why did this request take four seconds" stops being a mystery and becomes a picture. But a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it. Here is the moment it clicked. In the Trace Explorer I typed: gen ai.request.model = 'qwen3-coder-plus' Two things happened. First, the autocomplete suggested qwen3-coder-plus before I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system. Stop and consider what that means for agent debugging. I never told SigNoz what gen ai.request.model is. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary becomes the product's vocabulary . For LLM systems, where all the interesting facts live in custom attributes model, provider, token counts, agent role, tool name , this is the difference between an observability tool that fits and one you fight with. And the same query keeps working as you move across signals: trace id = '