Built for the Agents of SigNoz hackathon — Track 1, AI & Agent Observability.
Repo:[https://github.com/Eshan276/signoz_hackathon]
Install:curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh
The hackathon brief names it perfectly:
AI agents are chaining LLM calls, invoking tools, hitting vector DBs, and making decisions autonomously. But when latency spikes, costs explode, or an agent hallucinates in production, you're flying blind. You can't debug what you can't see.
Here's the thing though — the telemetry to not fly blind already exists. OpenTelemetry, OpenLLMetry, SigNoz's LLM dashboards: the pieces are all there. The reason most teams stay blind isn't that it's impossible. It's that wiring it all up is a weekend of yak-shaving: auto-instrumentation for each language, LLM and vector-DB spans, a collector, cost math, dashboards — all before the first span shows up.
So I built ** signoz-init**: a Go CLI that turns that weekend into
signoz-init init .
Point it at any Docker Compose stack. It detects each service, wires up full OpenTelemetry into SigNoz — HTTP, database, LLM and vector-DB spans with token counts and per-request cost in dollars — and confirms the telemetry actually arrived. Without touching your docker-compose.yml
or a single line of application code.
The pitch: observability should be a git diff, not a weekend.
Here's the trace from a single request through a RAG service, produced with zero application code changes:
web:POST /ask ← Node/Express gateway
└─ api:POST /ask ← FastAPI (root server span)
├─ api:qdrant.search ← vector DB
└─ api:chat gpt-4o-mini ← LLM call
├─ gen_ai.usage.input_tokens : 77
├─ gen_ai.usage.output_tokens: 25
└─ gen_ai.usage.cost_usd : 0.0000856 ← in dollars, not tokens
(Insert screenshot: the trace waterfall in SigNoz's Traces explorer, LLM span expanded.)
That last line is the whole idea. SigNoz tracks token counts natively, but it does not compute dollar cost — its own sample dashboards scale tokens by hand. signoz-init
emits gen_ai.usage.cost_usd
from an editable pricing table, filling a real gap.
The CLI runs five phases, and every one is something you can see and confirm:
| Phase | What happens |
|---|---|
| Detect | |
Parses docker-compose.yml , classifies each service by layered signals — build-context manifests → image name → command → ports. Reports unknown honestly rather than guessing. |
|
| Confirm | |
| Shows a detection table. Low-confidence guesses are flagged, not stated as fact. | |
| Generate | |
Writes docker-compose.override.yml + .signoz/ assets. Shows a full diff first. |
|
| Apply | |
| Rebuilds and restarts the stack. | |
| Verify | |
| Polls SigNoz until your services report, then prints what actually arrived. |
That last phase is what makes it a product rather than a YAML generator. Anything can write config and hope. Confirming telemetry landed is the difference:
✓ 2 services reporting, 168 spans
api 123 spans
POST /ask, qdrant.search, chat gpt-4o-mini
web 45 spans
POST /ask, tcp.connect
7 LLM spans with token counts
7 spans with cost attribution
The magic is getting instrumentation into a running container without editing code:
NODE_OPTIONS=--require /otel/otel-bootstrap.js
plus env vars, all in the override file.sitecustomize.py
on PYTHONPATH
. Python auto-imports sitecustomize
at interpreter startup, Everything lands in docker-compose.override.yml
, which Compose merges automatically. Reverting the entire thing is a single rm docker-compose.override.yml
.
SigNoz is the observability backend the whole tool targets, and I leaned on it at every layer:
Install via Foundry. SigNoz self-hosts through foundryctl
and an 8-line casting.yaml
(committed in the repo, so judges can re-run it). UI on :8080
, OTLP on :4317
/:4318
, with its own bundled OTel collector.
Traces. Every instrumented request becomes a distributed trace in SigNoz spanning web → api → qdrant → LLM
, viewable as a full waterfall in the Traces explorer.
Cost & tokens. By emitting canonical gen_ai.*
attributes, SigNoz's built-in LLM views and Cost Meter light up for free — and I ship a custom 12-widget dashboard imported through SigNoz's dashboards API: cost over time, tokens by model, cost by service, LLM p95 latency, vector-search latency.
(Insert screenshot: the LLM Cost dashboard.)
Verification. The CLI queries SigNoz to prove spans landed — ClickHouse directly for self-hosted (no credentials needed), or the /api/v2/services
API for SigNoz Cloud.
The single most valuable thing I learned: "SigNoz is running" is not the same as "SigNoz can accept telemetry."
Freshly cast, SigNoz's ingester logs cannot create agent without orgId
and never opens its OTLP ports — because the OpAMP server won't hand it a config until an organization exists, which only happens after first-run signup. Connections are refused, curl
returns exit 56, and it looks exactly like a networking bug. It isn't. You just have to register the first admin, and OTLP opens within ~30 seconds.
This is a product requirement, not just my local workaround. So signoz-init
detects the no-org state and guides you through it — because otherwise the first thing a new user sees is a silent black hole.
Cost and latency tell you that something is wrong, not what. So I added two signals that directly attack the "an agent hallucinates in production" line from the brief — signals auto-instrumentation fundamentally cannot produce, because only the application knows them.
gen_ai.response.groundedness
is a 0–1 score of how much of the model's answer actually came from the retrieved context. It's a deliberately cheap lexical heuristic — no second LLM call, no API key, no added latency — but it catches the failure that matters: an answer that wandered off the retrieved chunks.
def groundedness(answer, sources):
answer_tokens = meaningful_words(answer)
source_tokens = union(meaningful_words(s) for s in sources)
return len(answer_tokens & source_tokens) / len(answer_tokens)
A grounded answer scores ~1.0; an answer drawing on parametric knowledge scores ~0. It rides the same trace as the LLM span and rolls up natively in SigNoz — average groundedness, groundedness over time, the works.
session.id
on every span means cost, latency, and groundedness aggregate per conversation, not just per call. That turns "this request cost $0.00004" into "this conversation cost $0.40" — the number a budget owner actually cares about.
This one taught me a subtle lesson (see gotcha #10 below): cost lives on the LLM span, but the session id starts on the request span. To make "cost by conversation" work, a span processor has to stamp the session onto every span in the request. I only caught that the dashboard widget was silently empty because I tested the actual query, not just the happy path.
This is the part I'm proudest of, and the best evidence the tool is worth building — because a normal developer hits these and gives up. Every one is verified, hands-on:
TRACELOOP_BASE_URL
routes traces, not the api_endpoint
kwarg.Traceloop.init(api_endpoint=...)
alone exports nowhere, silently.async def
run_in_threadpool
loses OTel contextdef
endpoint works, because Starlette copies contextvars. Counter-intuitive, verified both ways.SpanProcessor.on_end
BatchSpanProcessor.span_exporter
is read-only._batch_processor._exporter
.COPY
source in, so docker compose restart
runs stale code. Several confusing results traced back to this.search
but NOT query_points
trace.get_tracer()
returns a proxy and every hand-written span is a NonRecordingSpan
that silently vanishes. Instrumented libraries emit spans while your own code emits nothing, with no error.Every one of these is now documented and handled in the tool, so its users never have to learn them.
curl -fsSL https://raw.githubusercontent.com/Eshan276/signoz_hackathon/main/install.sh | sh
signoz-init init ./demo
The demo is a real RAG stack — Node gateway → FastAPI → Qdrant → LLM — and it runs with or without an API key. No key uses a mock LLM that still emits proper gen_ai.*
spans with realistic tokens, so anyone can reproduce the full pipeline, cost included. With a key it calls a real model — and because it uses the OpenAI SDK, it works against any OpenAI-compatible endpoint (I verified it against Gemini via a single LLM_BASE_URL
change).
~2,600 lines of Go, 16 tests green, one command from blind to observable.
Observability should be a git diff
, not a weekend.
Built with SigNoz for the Agents of SigNoz hackathon. Repo: https://github.com/Eshan276/signoz_hackathon