# I Traced a Multi-Step LLM Agent With Self-Hosted SigNoz. One Feature Sold Me.

> Source: <https://dev.to/himanshu_748/i-traced-a-multi-step-llm-agent-with-self-hosted-signoz-one-feature-sold-me-4k71>
> Published: 2026-07-11 08:22:06+00:00

Multi-step LLM agents fail in a way normal backends don't. Nothing crashes. The pipeline "works", the answer is just bad, slow or three times more expensive than yesterday. `print()`

debugging tells you nothing, because the interesting question is never "did step 3 run". It is "what did step 3 see, which model actually answered and what did it cost".

So I self-hosted SigNoz and pointed a simulated agent pipeline at it: a four-step research assistant (plan, retrieve, generate, synthesize) instrumented with OpenTelemetry, emitting traces, metrics and logs, with GenAI semantic-convention attributes (`gen_ai.request.model`

, `gen_ai.usage.input_tokens`

and friends) on every LLM call.

This post is about the feature that turned out to be the most useful. It was not the one I expected.

Self-hosting used to mean wrangling a long docker-compose file. SigNoz now ships **Foundry**, a small CLI that casts the whole stack:

```
curl -fsSL https://signoz.io/foundry.sh | bash   # installs foundryctl (checksum-verified)
foundryctl cast -f casting.yaml                  # deploys the full stack on Docker
```

with a `casting.yaml`

that is all of eight lines:

```
apiVersion: v1alpha1
kind: Installation
metadata:
  name: signoz
spec:
  deployment:
    flavor: compose
    mode: docker
```

A few minutes of image pulls later, ClickHouse, Postgres, the SigNoz backend and an OTel collector were running. The UI is on `localhost:8080`

and the collector listens on `4317`

(gRPC) and `4318`

(HTTP).

**One gotcha worth knowing:** telemetry is rejected until you create the admin account in the UI. The collector registers with the backend over OpAMP, and until an organization exists the backend answers "cannot create agent without orgId" and the OTLP ports reset every connection. If your exporter logs `Connection reset by peer`

on a fresh install, you haven't finished the two-minute signup at `localhost:8080`

yet. Create the account and ingestion starts working within about thirty seconds, no restarts needed.

My demo app needed zero SigNoz-specific code: the stock OpenTelemetry SDK exporting OTLP to `localhost:4318`

. That is the point of an OTel-native backend. There is no vendor agent, so nothing about the app knows SigNoz exists.

Each simulated request produces one trace:

```
agent.request                    (root, 2.02s)
├── agent.plan                   (850ms)
│   └── gen_ai.generate plan     (696ms, gen_ai.* attributes)
├── agent.retrieve               (461ms, sometimes errors: vector store timeout)
├── agent.generate               (564ms)
│   └── gen_ai.generate answer   (564ms, gen_ai.* attributes)
└── agent.synthesize             (139ms, sometimes errors: citation validation)
```

Every `gen_ai.generate`

span carries the GenAI semantic conventions: operation name, provider, model, input tokens and output tokens. Alongside the traces the app emits counters for token usage (`gen_ai.client.token.usage`

) and estimated spend (`agent.llm.cost`

), both tagged by model and provider, a request-duration histogram and structured logs that inherit the active trace context automatically.

Sixty simulated requests later: 420 spans, 134 logs and a few hundred metric samples, all visible in the UI. The Services page picked up `research-assistant`

on its own with RED metrics already computed (p99 latency, error rate, throughput). I wrote no configuration for that.

And it is genuinely good. The trace detail view renders a flame graph and waterfall that read exactly like the agent's mental model: plan, then retrieve, then generate, then synthesize, with the LLM calls nested inside the steps that made them. Clicking any `gen_ai.generate`

span opens a details panel with every attribute I set: model `llama-4-maverick`

, provider `meta`

, 1501 input tokens, 90 output tokens, plus a percentile badge telling me this span sat at p10 of its peers. "Why did this request take four seconds" stops being a mystery and becomes a picture.

But a pretty waterfall is table stakes for a tracing tool. The thing that sold me was what happens around it.

Here is the moment it clicked. In the Trace Explorer I typed:

```
gen_ai.request.model = 'qwen3-coder-plus'
```

Two things happened. First, the autocomplete suggested `qwen3-coder-plus`

before I finished typing, because SigNoz had already indexed the values of an attribute I invented an hour earlier. Second, the results came back instantly: only the LLM spans that were served by that model, across every trace in the system.

Stop and consider what that means for agent debugging. I never told SigNoz what `gen_ai.request.model`

is. There is no schema registration, no field mapping, no config file. Any attribute your instrumentation emits is immediately a first-class, autocompleted, indexed query dimension. Your instrumentation vocabulary *becomes the product's vocabulary*. For LLM systems, where all the interesting facts live in custom attributes (model, provider, token counts, agent role, tool name), this is the difference between an observability tool that fits and one you fight with.

And the same query keeps working as you move across signals:

`trace_id = '<this trace>'`

pre-filled and the time window auto-scoped. I got back exactly the two log lines belonging to that request, correlated purely by the trace context the OTel logging handler injects. Nobody parses log lines to find a request id. The correlation is structural.`gen_ai.client.token.usage`

showing 6 time series (3 models times 2 token types), `agent.llm.cost`

in usd showing 3. Cost per model is a group-by away, using the exact attribute names from my instrumentation.Most observability stacks treat custom attributes as second-class blobs that need schema work before they are queryable. SigNoz treats them as the whole point. For agent systems, they are.

Going deep on one feature meant walking past a lot of others. Quick notes from the tour, agent-flavored:

Each of these uses the attribute vocabulary your instrumentation defines. That is the theme of the whole product, and it is why the one feature I picked is really the foundation the rest stand on.

The uncomfortable truth about multi-agent LLM systems is that their most important behavior (which model ran, what it consumed, what it cost, why it was retried) is invisible to conventional monitoring. It lives entirely in domain-specific span attributes. A backend that makes those attributes instantly queryable, correlatable across traces and logs, and promotable into dashboards and alerts is not a nice-to-have there. It is the debugger.

The whole experiment cost me an afternoon: one CLI install, eight lines of YAML, a stock OTel SDK and zero vendor code in the app. I'm taking this setup into the Agents of SigNoz hackathon (July 20 to 26), where the plan is considerably less simulated. If you're building anything agent-shaped, self-host SigNoz and type one of your own attribute names into the Trace Explorer. That autocomplete dropdown is the moment you'll get it too.

*The demo app is a 170-line Python script using only opentelemetry-sdk and the OTLP HTTP exporter. Stack: SigNoz self-hosted via Foundry on Docker (ClickHouse + Postgres + OTel collector), macOS host.*

*Demo code, self-host config and blog source: github.com/himanshu748/signoz-agent-observability. Team 404Found.*