# Observability should be a `git diff`, not a weekend: instrumenting an AI app with one command using SigNoz > Source: > Published: 2026-07-26 00:48:58+00:00 *Built for the Agents of SigNoz hackathon — Track 1, AI & Agent Observability.* Repo:[https://github.com/Eshan276/signoz_hackathon] Install:`curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh` The hackathon brief names it perfectly: AI agents are chaining LLM calls, invoking tools, hitting vector DBs, and making decisions autonomously. But when latency spikes, costs explode, or an agent hallucinates in production, you're flying blind. You can't debug what you can't see. Here's the thing though — the telemetry to *not* fly blind already exists. OpenTelemetry, OpenLLMetry, SigNoz's LLM dashboards: the pieces are all there. The reason most teams stay blind isn't that it's impossible. It's that wiring it all up is a *weekend* of yak-shaving: auto-instrumentation for each language, LLM and vector-DB spans, a collector, cost math, dashboards — all before the first span shows up. So I built ** signoz-init**: a Go CLI that turns that weekend into ``` signoz-init init . ``` Point it at any Docker Compose stack. It detects each service, wires up full OpenTelemetry into SigNoz — HTTP, database, **LLM and vector-DB spans with token counts and per-request cost in dollars** — and confirms the telemetry actually arrived. Without touching your `docker-compose.yml` or a single line of application code. The pitch: **observability should be a git diff, not a weekend.** Here's the trace from a single request through a RAG service, produced with **zero application code changes**: ``` web:POST /ask ← Node/Express gateway └─ api:POST /ask ← FastAPI (root server span) ├─ api:qdrant.search ← vector DB └─ api:chat gpt-4o-mini ← LLM call ├─ gen_ai.usage.input_tokens : 77 ├─ gen_ai.usage.output_tokens: 25 └─ gen_ai.usage.cost_usd : 0.0000856 ← in dollars, not tokens ``` *(Insert screenshot: the trace waterfall in SigNoz's Traces explorer, LLM span expanded.)* That last line is the whole idea. **SigNoz tracks token counts natively, but it does not compute dollar cost** — its own sample dashboards scale tokens by hand. `signoz-init` emits `gen_ai.usage.cost_usd` from an editable pricing table, filling a real gap. The CLI runs five phases, and every one is something you can see and confirm: | Phase | What happens | |---|---| Detect | Parses `docker-compose.yml` , classifies each service by layered signals — build-context manifests → image name → command → ports. Reports `unknown` honestly rather than guessing. | Confirm | Shows a detection table. Low-confidence guesses are flagged, not stated as fact. | Generate | Writes `docker-compose.override.yml` + `.signoz/` assets. Shows a full diff first. | Apply | Rebuilds and restarts the stack. | Verify | Polls SigNoz until your services report, then prints what actually arrived. | That last phase is what makes it a *product* rather than a YAML generator. Anything can write config and hope. Confirming telemetry landed is the difference: ``` ✓ 2 services reporting, 168 spans api 123 spans POST /ask, qdrant.search, chat gpt-4o-mini web 45 spans POST /ask, tcp.connect 7 LLM spans with token counts 7 spans with cost attribution ``` The magic is getting instrumentation into a running container without editing code: `NODE_OPTIONS=--require /otel/otel-bootstrap.js` plus env vars, all in the override file.`sitecustomize.py` on `PYTHONPATH` . Python auto-imports `sitecustomize` at interpreter startup, Everything lands in `docker-compose.override.yml` , which Compose merges automatically. Reverting the entire thing is a single `rm docker-compose.override.yml` . SigNoz is the observability backend the whole tool targets, and I leaned on it at every layer: **Install via Foundry.** SigNoz self-hosts through `foundryctl` and an 8-line `casting.yaml` (committed in the repo, so judges can re-run it). UI on `:8080` , OTLP on `:4317` /`:4318` , with its own bundled OTel collector. **Traces.** Every instrumented request becomes a distributed trace in SigNoz spanning `web → api → qdrant → LLM` , viewable as a full waterfall in the Traces explorer. **Cost & tokens.** By emitting canonical `gen_ai.*` attributes, SigNoz's built-in LLM views and Cost Meter light up for free — and I ship a custom 12-widget dashboard imported through SigNoz's dashboards API: cost over time, tokens by model, cost by service, LLM p95 latency, vector-search latency. *(Insert screenshot: the LLM Cost dashboard.)* **Verification.** The CLI queries SigNoz to prove spans landed — ClickHouse directly for self-hosted (no credentials needed), or the `/api/v2/services` API for SigNoz Cloud. The single most valuable thing I learned: **"SigNoz is running" is not the same as "SigNoz can accept telemetry."** Freshly cast, SigNoz's ingester logs `cannot create agent without orgId` and **never opens its OTLP ports** — because the OpAMP server won't hand it a config until an organization exists, which only happens after first-run signup. Connections are *refused*, `curl` returns exit 56, and it looks exactly like a networking bug. It isn't. You just have to register the first admin, and OTLP opens within ~30 seconds. This is a product requirement, not just my local workaround. So `signoz-init` detects the no-org state and guides you through it — because otherwise the first thing a new user sees is a silent black hole. Cost and latency tell you *that* something is wrong, not *what*. So I added two signals that directly attack the "an agent hallucinates in production" line from the brief — signals **auto-instrumentation fundamentally cannot produce, because only the application knows them.** `gen_ai.response.groundedness` is a 0–1 score of how much of the model's answer actually came from the retrieved context. It's a deliberately cheap lexical heuristic — **no second LLM call, no API key, no added latency** — but it catches the failure that matters: an answer that wandered off the retrieved chunks. ``` python def groundedness(answer, sources): answer_tokens = meaningful_words(answer) source_tokens = union(meaningful_words(s) for s in sources) return len(answer_tokens & source_tokens) / len(answer_tokens) ``` A grounded answer scores ~1.0; an answer drawing on parametric knowledge scores ~0. It rides the same trace as the LLM span and rolls up natively in SigNoz — average groundedness, groundedness over time, the works. `session.id` on every span means cost, latency, and groundedness aggregate **per conversation**, not just per call. That turns "this request cost $0.00004" into "this *conversation* cost $0.40" — the number a budget owner actually cares about. This one taught me a subtle lesson (see gotcha #10 below): cost lives on the *LLM* span, but the session id starts on the *request* span. To make "cost by conversation" work, a span processor has to stamp the session onto *every* span in the request. I only caught that the dashboard widget was silently empty because I tested the actual query, not just the happy path. This is the part I'm proudest of, and the best evidence the tool is worth building — because a normal developer hits these and gives up. Every one is verified, hands-on: `TRACELOOP_BASE_URL` routes traces, not the `api_endpoint` kwarg.`Traceloop.init(api_endpoint=...)` alone exports nowhere, silently.`async def` + `run_in_threadpool` loses OTel context`def` endpoint works, because Starlette copies contextvars. Counter-intuitive, verified both ways.`SpanProcessor.on_end` `BatchSpanProcessor.span_exporter` is read-only.`_batch_processor._exporter` .`COPY` source in, so `docker compose restart` runs stale code. Several confusing results traced back to this.`search` but NOT `query_points` `trace.get_tracer()` returns a proxy and every hand-written span is a `NonRecordingSpan` that silently vanishes. Instrumented libraries emit spans while your own code emits nothing, with no error.Every one of these is now documented and handled in the tool, so its users never have to learn them. ``` # install (no Go needed — prebuilt binary) curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh # stand up SigNoz once (foundryctl), then: signoz-init init ./demo ``` The demo is a real RAG stack — Node gateway → FastAPI → Qdrant → LLM — and it **runs with or without an API key.** No key uses a mock LLM that still emits proper `gen_ai.*` spans with realistic tokens, so anyone can reproduce the full pipeline, cost included. With a key it calls a real model — and because it uses the OpenAI SDK, it works against any OpenAI-compatible endpoint (I verified it against Gemini via a single `LLM_BASE_URL` change). **~2,600 lines of Go, 16 tests green, one command from blind to observable.** Observability should be a `git diff` , not a weekend. *Built with SigNoz for the Agents of SigNoz hackathon. Repo: https://github.com/Eshan276/signoz_hackathon*