A warm-up with SigNoz, before the "Agents of SigNoz" hackathon proper
I'm building an AI-powered platform to help patients navigate rare and undiagnosed disease workups — routing them toward the right specialists and India's government-designated Centres of Excellence instead of years of misdiagnosis. The architecture has been frozen for a while now: a LangGraph-based multi-agent system with twelve components — an Orchestrator, a Symptom Intake Agent, an HPO Extraction Agent, a Clinical Triage Engine, a RAG Retrieval Engine, a Differential Reasoning Agent, a CoE Navigation Agent, and a few more, all passing state to each other, all calling out to LLM APIs, a vector store, and Postgres.
Here's the uncomfortable truth I ran into while finishing the documentation: I could describe every agent's responsibility in detail, in a Word document, with data flow diagrams. What I couldn't tell you was how long the HPO Extraction Agent actually takes when the RAG Retrieval Engine is slow, or whether a failed differential reasoning call fails loudly or just silently returns an empty list that the next agent quietly accepts.
That gap — between "I designed this system" and "I can see this system while it runs" — is exactly what the WeMakeDevs × SigNoz warm-up challenge is built around. So before the actual hackathon sprint (Track 01, AI & Agent Observability, is the obvious fit for my project), I self-hosted SigNoz and wired up an instrumented slice of my pipeline to see what "flying blind," a phrase the hackathon page uses almost accusingly, actually feels like — and what the alternative feels like.
Scope note:this post covers a representative two-service slice of the full pipeline (an API gateway that mimics the Orchestrator, plus one downstream agent), not the entire twelve-component system. The goal here is to genuinely understand SigNoz's traces/metrics/logs/dashboards/alerts before the real build, not to pretend I shipped the whole platform in a weekend.
Multi-agent systems have a specific failure mode that single-service apps don't: the bug is rarely in the code you're looking at. If the Differential Reasoning Agent returns a bad answer, is it because:
With print()
statements and scattered log files, answering that means grepping through however many terminals you have open, lining up timestamps by eye, and hoping the clocks agree. That's not a debugging process, it's archaeology. And it gets exponentially worse the moment you add a second agent, a database, and an external API — which describes almost every serious AI system being built for this hackathon.
It's tempting to think of these as three flavors of the same thing. They're not; they answer different questions, and observability tools only get powerful when they can correlate across all three.
abc123
". Great for the For a multi-agent pipeline specifically, traces matter more than they do in a typical CRUD app, because the "request" isn't one HTTP call — it's a chain of LLM calls, tool invocations, and retrieval hops, and the only way to see that chain as one thing is distributed tracing with correctly propagated context.
There are plenty of observability tools. What made SigNoz the right fit for this specific problem:
SigNoz's self-host install has changed recently — if you've seen an older tutorial reference a docker-compose up
command straight out of the SigNoz repo's deploy/
folder, that path has been deprecated in a recent release. The currently supported installer is Foundry, a declarative CLI that generates and manages the Compose (or Kubernetes) files for you.
The actual flow is refreshingly small:
curl -fsSL https://signoz.io/foundry.sh | bash
cat > casting.yaml << 'EOF'
apiVersion: v1alpha1
kind: Installation
metadata:
name: signoz
spec:
deployment:
flavor: compose
mode: docker
EOF
foundryctl cast -f casting.yaml
cast
isn't one opaque command — it chains three stages (gauge
validates your Docker setup, forge
renders the actual Compose files into a pours/
directory, cast
brings the stack up), and you can run each stage separately if you want to inspect the generated Compose file before trusting it with docker compose up
. That separation is a nice bit of design: I could see exactly what containers were about to start (ClickHouse for storage, a Postgres-backed metastore, a ClickHouse Keeper node, the OTel collector, and the SigNoz UI/API server itself) before committing.
Prerequisite check:Docker Engine 20.10+, the Compose v2 plugin, and at least 4GB of memory allocated to Docker. If containers keep restarting, that memory ceiling is the first thing to raise — SigNoz's docs call this out explicitly, and ClickHouse in particular is memory-hungry on a cold start.
Once docker ps
shows everything healthy, the UI comes up at http://localhost:8080
. Ports 4317
(OTLP gRPC) and 4318
(OTLP HTTP) are what your application will actually talk to.
Although the deployment itself was straightforward, the most useful confirmation came from seeing the services come up cleanly and the SigNoz UI become reachable locally. That moment confirmed the observability stack was ready to receive telemetry from my application.
Rather than spinning up a "hello world" Flask app just to say I instrumented something, I stood up a stripped-down FastAPI service that mirrors the shape of my real Symptom Intake Agent: it receives a symptom payload, calls out to a downstream "extraction" service (standing in for the HPO Extraction Agent), and returns a structured response. Two services, one synchronous hop between them — small enough to reason about, real enough to show actual cross-service tracing instead of a single-span demo.
SigNoz's Python guide recommends opentelemetry-distro
for zero-code auto-instrumentation, which detects your installed libraries and wraps them automatically:
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap --action=install
That second command is doing more than it looks like — it scans your installed packages (FastAPI, requests
, whatever database driver you're using) and installs the matching OpenTelemetry instrumentation library for each one. Run it after your app's dependencies are installed, or it has nothing to detect.
Environment variables point the SDK at self-hosted SigNoz instead of the cloud ingestion endpoint:
export OTEL_RESOURCE_ATTRIBUTES="service.name=symptom-intake-agent,service.version=$(git rev-parse --short HEAD)"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_METRICS_EXPORTER="none" # traces first, metrics deliberately off for now
Callout — why disable metrics here:opentelemetry-distro
enables metrics by default, and the auto-instrumentation for HTTP clients likerequests
andhttpx
will start emitting a duration histogram forevery outgoing callthe moment metrics are on. For a chatty agent pipeline making several LLM and retrieval calls per request, that's a lot of unplanned data volume before you've even decided you want it. Turning it off at first keeps the traces experiment clean; flipping it tootlp
later is a one-line change.
Then the run command changes from uvicorn main:app
to:
opentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000
The gotcha that would have cost me an hour if I hadn't read the docs first: opentelemetry-instrument
does not support Uvicorn's --workers
flag, and --reload
outright breaks instrumentation for any framework, not just Uvicorn. The reason is unglamorous but important — the OpenTelemetry SDK's batch span processor runs on a background thread, and Python's fork model doesn't safely carry that thread into forked worker processes unless the server explicitly re-initializes it after fork. Uvicorn's worker mode doesn't do that; Gunicorn ( the app fresh in each forked worker) does. So for multi-worker production instrumentation, the documented pattern is Gunicorn with Uvicorn workers:
opentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:8000
For a single-worker dev instance, plain opentelemetry-instrument uvicorn
without --reload
is fine, and that's what I used for this warm-up.
One lesson I took away during setup is that instrumentation deserves the same attention as application code. A small configuration mistake can prevent telemetry from appearing, so validating the OpenTelemetry configuration early saves significant debugging time later.
To make the tracing story concrete, I sent a single synthetic symptom payload through the two-service slice — a POST /intake
call carrying a small set of HPO-style symptom terms — and watched it end to end instead of looking at aggregate graphs first.
Conceptually, the request's journey looks like this:
Client
│
▼
symptom-intake-agent (FastAPI, span: POST /intake)
│ validates payload
│ calls downstream extraction service (span: HTTP POST /extract)
▼
extraction-service (FastAPI, span: POST /extract)
│ normalizes symptom terms
│ returns structured HPO-like terms
▲
symptom-intake-agent (assembles response)
│
▼
Client
Because both services are auto-instrumented with OpenTelemetry and both export to the same SigNoz collector, the HTTP call between them isn't two disconnected traces — it's one trace with a parent span in symptom-intake-agent
and a child span in extraction-service
, stitched together by W3C trace-context propagation happening automatically inside the instrumented requests
/httpx
client and the receiving FastAPI middleware. That stitching is the entire point of distributed tracing, and it's also the thing that's genuinely hard to bolt on after the fact with hand-rolled logging.
Opening the Traces tab in SigNoz and drilling into that request shows a flamegraph / Gantt-style view: the parent span for POST /intake
as a wide bar at the top, and the nested POST /extract
call as a shorter bar underneath it, positioned exactly where in time it started and ended relative to the parent.
This is a small example, but it points at something I wouldn't have appreciated without seeing it laid out visually: in a chain of agents, latency is additive and attribution is not. If the whole request takes 900ms and the extraction call alone accounts for 700 of them, that's not a fact you can eyeball from logs with two separate timestamps in two separate files — you'd have to manually subtract wall-clock times and hope your service clocks are synced. In the trace view it's just... the size of the bar.
For the real twelve-agent pipeline, this is the difference between "the whole request is slow" (unhelpful) and "the RAG Retrieval Engine is 60% of total latency on differential-diagnosis-heavy queries" (actionable, and exactly the kind of finding that should shape which agent gets optimized first).
Watching the request appear as a complete parent–child trace reinforced the value of distributed tracing. Rather than viewing isolated events, I could understand the entire request lifecycle as one connected execution flow.
A single trace tells you about a single request. Metrics tell you whether that request was typical or an outlier. SigNoz auto-generates APM dashboards from the trace data your services already send — p99 latency, error rate, throughput, and Apdex score per service — without any separate metrics instrumentation required for basic request-level numbers.
That auto-generation matters for a hackathon timeline: you get a "is this service healthy" dashboard for free the moment traces are flowing, before you've written a single custom metric. For anything beyond that — say, tracking token counts per LLM call, or queue depth in an agent's task backlog — you'd add manual metrics instrumentation (counters, histograms) via the OpenTelemetry Python metrics API, which is the natural "next step" once the free APM dashboard stops being enough.
The reason logs still matter even with rich traces: a span tells you that something happened in that time window and how long it took, but it doesn't automatically carry your application's own log lines — "symptom set had 3 unrecognized terms," say. SigNoz's log pipeline is built to correlate logs with the trace and span ID active at the time the log line was emitted, so from a slow span you can jump to exactly the log lines produced during that span, instead of grepping a log file by approximate timestamp.
For an agent pipeline where a "failure" is often silent — an agent returning an empty or low-confidence result rather than throwing an exception — this correlation is what turns "the trace shows this span took long" into "and here's the log line explaining that it was retrying against a rate-limited LLM endpoint three times before it got a response."
Traces and logs answer "what happened in this one request." Dashboards and alerts answer "should I be watching this at all right now." Once telemetry is flowing, the natural next step is a small custom dashboard — p99 latency per agent, error rate per agent, and a panel showing the downstream-call duration breakdown — built with SigNoz's Query Builder, which lets you compose these panels without hand-writing ClickHouse SQL, while still dropping into raw queries if you need something the builder doesn't expose yet.
Alerts close the loop: a rule on error rate or p99 latency crossing a threshold means you find out about a regressing agent from a notification, not from a user complaint or a judge poking at your demo. For a system that's explicitly not meant to give numeric diagnostic confidence scores to patients but is meant to route them reliably, an alert on the Clinical Triage Engine's error rate is arguably more important than almost any other dashboard panel in the whole project.
After exploring dashboards and alerting, it became clear that observability is not only about debugging existing issues but also about identifying performance regressions before they become user-facing problems.
If I had to choose one capability that stood out during this warm-up, it would be distributed tracing. Before exploring SigNoz, I tended to think about debugging as collecting logs from different parts of an application and manually piecing events together. Tracing changes that workflow entirely by presenting a request as a single connected journey across services.
What I appreciated most is that tracing provides context rather than isolated events. Instead of simply knowing that a request was slower than expected, I could understand which stage contributed to the overall execution time and how the different services interacted. For applications built from multiple components—especially AI and agentic systems—this end-to-end visibility is far more valuable than reading individual log entries in isolation.
As I continue building my rare disease navigation platform, distributed tracing is the feature I expect to rely on the most. It gives me confidence that as more agents and external services are added, I will still be able to understand the behaviour of the system without turning debugging into guesswork.
One thing that initially took time to understand was how logs, metrics, and traces complement rather than replace one another. I first approached them as different views of the same information, but using SigNoz made it clear that each answers a different question: metrics reveal trends, logs provide detailed events, and traces explain the lifecycle of an individual request.
The biggest insight for me was recognizing that observability is something to build into an application from the beginning rather than adding only when problems appear. Once telemetry is available, understanding system behaviour becomes significantly easier because every request carries useful operational context alongside the application logic.
If I could give my past self one piece of advice, it would be simple: treat observability as part of the development process, not as a finishing step. That mindset will be even more valuable as I build the complete AI-agent workflow for the hackathon.
--reload
are real and documented for a reason — hit them once, and you'll remember the fix forever.My project's whole premise is routing patients through a system they can't see — different agents, different confidence signals, a government CoE network behind it all. It would be a strange kind of hypocrisy to build that system and then have zero visibility into how my own pipeline behaves in production. This warm-up convinced me that wiring up SigNoz from day one of the real build — not as an afterthought before a demo — is the right call, because the moment I add agent nine through twelve, tracing is the only thing that will keep "which component broke" from turning back into archaeology.
If you're building anything with more than one moving part for this hackathon — agents calling agents, services calling services — do the same warm-up before your sprint clock starts. It's a few hours now against a much worse debugging session later.
Built for the WeMakeDevs × SigNoz "Agents of SigNoz" hackathon warm-up blog challenge. Self-hosted SigNoz via Foundry, OpenTelemetry auto-instrumentation on FastAPI, submitted ahead of the July 19, 2026 deadline.