Observability should be a `git diff`, not a weekend: instrumenting an AI app with one command using SigNoz

A developer built signoz-init, a Go CLI that instruments AI applications with full OpenTelemetry observability into SigNoz using a single command, without modifying application code. The tool detects services in a Docker Compose stack, wires up LLM and vector-DB spans with token counts and per-request cost in dollars, and verifies telemetry delivery. 'Observability should be a git diff, not a weekend,' the developer said.

Built for the Agents of SigNoz hackathon — Track 1, AI & Agent Observability. Repo: https://github.com/Eshan276/signoz hackathon Install: curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz hackathon/main/install.sh | sh The hackathon brief names it perfectly: AI agents are chaining LLM calls, invoking tools, hitting vector DBs, and making decisions autonomously. But when latency spikes, costs explode, or an agent hallucinates in production, you're flying blind. You can't debug what you can't see. Here's the thing though — the telemetry to not fly blind already exists. OpenTelemetry, OpenLLMetry, SigNoz's LLM dashboards: the pieces are all there. The reason most teams stay blind isn't that it's impossible. It's that wiring it all up is a weekend of yak-shaving: auto-instrumentation for each language, LLM and vector-DB spans, a collector, cost math, dashboards — all before the first span shows up. So I built signoz-init : a Go CLI that turns that weekend into signoz-init init . Point it at any Docker Compose stack. It detects each service, wires up full OpenTelemetry into SigNoz — HTTP, database, LLM and vector-DB spans with token counts and per-request cost in dollars — and confirms the telemetry actually arrived. Without touching your docker-compose.yml or a single line of application code. The pitch: observability should be a git diff, not a weekend. Here's the trace from a single request through a RAG service, produced with zero application code changes : web:POST /ask ← Node/Express gateway └─ api:POST /ask ← FastAPI root server span ├─ api:qdrant.search ← vector DB └─ api:chat gpt-4o-mini ← LLM call ├─ gen ai.usage.input tokens : 77 ├─ gen ai.usage.output tokens: 25 └─ gen ai.usage.cost usd : 0.0000856 ← in dollars, not tokens Insert screenshot: the trace waterfall in SigNoz's Traces explorer, LLM span expanded. That last line is the whole idea. SigNoz tracks token counts natively, but it does not compute dollar cost — its own sample dashboards scale tokens by hand. signoz-init emits gen ai.usage.cost usd from an editable pricing table, filling a real gap. The CLI runs five phases, and every one is something you can see and confirm: | Phase | What happens | |---|---| Detect | Parses docker-compose.yml , classifies each service by layered signals — build-context manifests → image name → command → ports. Reports unknown honestly rather than guessing. | Confirm | Shows a detection table. Low-confidence guesses are flagged, not stated as fact. | Generate | Writes docker-compose.override.yml + .signoz/ assets. Shows a full diff first. | Apply | Rebuilds and restarts the stack. | Verify | Polls SigNoz until your services report, then prints what actually arrived. | That last phase is what makes it a product rather than a YAML generator. Anything can write config and hope. Confirming telemetry landed is the difference: ✓ 2 services reporting, 168 spans api 123 spans POST /ask, qdrant.search, chat gpt-4o-mini web 45 spans POST /ask, tcp.connect 7 LLM spans with token counts 7 spans with cost attribution The magic is getting instrumentation into a running container without editing code: NODE OPTIONS=--require /otel/otel-bootstrap.js plus env vars, all in the override file. sitecustomize.py on PYTHONPATH . Python auto-imports sitecustomize at interpreter startup, Everything lands in docker-compose.override.yml , which Compose merges automatically. Reverting the entire thing is a single rm docker-compose.override.yml . SigNoz is the observability backend the whole tool targets, and I leaned on it at every layer: Install via Foundry. SigNoz self-hosts through foundryctl and an 8-line casting.yaml committed in the repo, so judges can re-run it . UI on :8080 , OTLP on :4317 / :4318 , with its own bundled OTel collector. Traces. Every instrumented request becomes a distributed trace in SigNoz spanning web → api → qdrant → LLM , viewable as a full waterfall in the Traces explorer. Cost & tokens. By emitting canonical gen ai. attributes, SigNoz's built-in LLM views and Cost Meter light up for free — and I ship a custom 12-widget dashboard imported through SigNoz's dashboards API: cost over time, tokens by model, cost by service, LLM p95 latency, vector-search latency. Insert screenshot: the LLM Cost dashboard. Verification. The CLI queries SigNoz to prove spans landed — ClickHouse directly for self-hosted no credentials needed , or the /api/v2/services API for SigNoz Cloud. The single most valuable thing I learned: "SigNoz is running" is not the same as "SigNoz can accept telemetry." Freshly cast, SigNoz's ingester logs cannot create agent without orgId and never opens its OTLP ports — because the OpAMP server won't hand it a config until an organization exists, which only happens after first-run signup. Connections are refused , curl returns exit 56, and it looks exactly like a networking bug. It isn't. You just have to register the first admin, and OTLP opens within ~30 seconds. This is a product requirement, not just my local workaround. So signoz-init detects the no-org state and guides you through it — because otherwise the first thing a new user sees is a silent black hole. Cost and latency tell you that something is wrong, not what . So I added two signals that directly attack the "an agent hallucinates in production" line from the brief — signals auto-instrumentation fundamentally cannot produce, because only the application knows them. gen ai.response.groundedness is a 0–1 score of how much of the model's answer actually came from the retrieved context. It's a deliberately cheap lexical heuristic — no second LLM call, no API key, no added latency — but it catches the failure that matters: an answer that wandered off the retrieved chunks. python def groundedness answer, sources : answer tokens = meaningful words answer source tokens = union meaningful words s for s in sources return len answer tokens & source tokens / len answer tokens A grounded answer scores ~1.0; an answer drawing on parametric knowledge scores ~0. It rides the same trace as the LLM span and rolls up natively in SigNoz — average groundedness, groundedness over time, the works. session.id on every span means cost, latency, and groundedness aggregate per conversation , not just per call. That turns "this request cost $0.00004" into "this conversation cost $0.40" — the number a budget owner actually cares about. This one taught me a subtle lesson see gotcha 10 below : cost lives on the LLM span, but the session id starts on the request span. To make "cost by conversation" work, a span processor has to stamp the session onto every span in the request. I only caught that the dashboard widget was silently empty because I tested the actual query, not just the happy path. This is the part I'm proudest of, and the best evidence the tool is worth building — because a normal developer hits these and gives up. Every one is verified, hands-on: TRACELOOP BASE URL routes traces, not the api endpoint kwarg. Traceloop.init api endpoint=... alone exports nowhere, silently. async def + run in threadpool loses OTel context def endpoint works, because Starlette copies contextvars. Counter-intuitive, verified both ways. SpanProcessor.on end BatchSpanProcessor.span exporter is read-only. batch processor. exporter . COPY source in, so docker compose restart runs stale code. Several confusing results traced back to this. search but NOT query points trace.get tracer returns a proxy and every hand-written span is a NonRecordingSpan that silently vanishes. Instrumented libraries emit spans while your own code emits nothing, with no error.Every one of these is now documented and handled in the tool, so its users never have to learn them. install no Go needed — prebuilt binary curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz hackathon/main/install.sh | sh stand up SigNoz once foundryctl , then: signoz-init init ./demo The demo is a real RAG stack — Node gateway → FastAPI → Qdrant → LLM — and it runs with or without an API key. No key uses a mock LLM that still emits proper gen ai. spans with realistic tokens, so anyone can reproduce the full pipeline, cost included. With a key it calls a real model — and because it uses the OpenAI SDK, it works against any OpenAI-compatible endpoint I verified it against Gemini via a single LLM BASE URL change . ~2,600 lines of Go, 16 tests green, one command from blind to observable. Observability should be a git diff , not a weekend. Built with SigNoz for the Agents of SigNoz hackathon. Repo: https://github.com/Eshan276/signoz hackathon