{"slug": "i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz", "title": "I Gave My AI Agent Pipeline a Nervous System with SigNoz", "summary": "A developer building an AI-powered platform for rare disease diagnosis used SigNoz to add observability to a LangGraph-based multi-agent system. The system, which includes twelve components like an Orchestrator and Symptom Intake Agent, suffered from opaque failures that traditional logging couldn't diagnose. By self-hosting SigNoz and instrumenting a slice of the pipeline, the developer gained distributed tracing, metrics, and logs to correlate across agent calls, LLM APIs, and databases.", "body_md": "*A warm-up with SigNoz, before the \"Agents of SigNoz\" hackathon proper*\n\nI'm building an AI-powered platform to help patients navigate rare and undiagnosed disease workups — routing them toward the right specialists and India's government-designated Centres of Excellence instead of years of misdiagnosis. The architecture has been frozen for a while now: a LangGraph-based multi-agent system with twelve components — an Orchestrator, a Symptom Intake Agent, an HPO Extraction Agent, a Clinical Triage Engine, a RAG Retrieval Engine, a Differential Reasoning Agent, a CoE Navigation Agent, and a few more, all passing state to each other, all calling out to LLM APIs, a vector store, and Postgres.\n\nHere's the uncomfortable truth I ran into while finishing the documentation: I could describe every agent's responsibility in detail, in a Word document, with data flow diagrams. What I *couldn't* tell you was how long the HPO Extraction Agent actually takes when the RAG Retrieval Engine is slow, or whether a failed differential reasoning call fails loudly or just silently returns an empty list that the next agent quietly accepts.\n\nThat gap — between \"I designed this system\" and \"I can see this system while it runs\" — is exactly what the WeMakeDevs × SigNoz warm-up challenge is built around. So before the actual hackathon sprint (Track 01, AI & Agent Observability, is the obvious fit for my project), I self-hosted SigNoz and wired up an instrumented slice of my pipeline to see what \"flying blind,\" a phrase the hackathon page uses almost accusingly, actually feels like — and what the alternative feels like.\n\nScope note:this post covers a representative two-service slice of the full pipeline (an API gateway that mimics the Orchestrator, plus one downstream agent), not the entire twelve-component system. The goal here is to genuinely understand SigNoz's traces/metrics/logs/dashboards/alerts before the real build, not to pretend I shipped the whole platform in a weekend.\n\nMulti-agent systems have a specific failure mode that single-service apps don't: **the bug is rarely in the code you're looking at.** If the Differential Reasoning Agent returns a bad answer, is it because:\n\nWith `print()`\n\nstatements and scattered log files, answering that means grepping through however many terminals you have open, lining up timestamps by eye, and hoping the clocks agree. That's not a debugging process, it's archaeology. And it gets exponentially worse the moment you add a second agent, a database, and an external API — which describes almost every serious AI system being built for this hackathon.\n\nIt's tempting to think of these as three flavors of the same thing. They're not; they answer different questions, and observability tools only get powerful when they can correlate across all three.\n\n`abc123`\n\n\". Great for the For a multi-agent pipeline specifically, traces matter more than they do in a typical CRUD app, because the \"request\" isn't one HTTP call — it's a chain of LLM calls, tool invocations, and retrieval hops, and the only way to see that chain as *one thing* is distributed tracing with correctly propagated context.\n\nThere are plenty of observability tools. What made SigNoz the right fit for this specific problem:\n\nSigNoz's self-host install has changed recently — if you've seen an older tutorial reference a `docker-compose up`\n\ncommand straight out of the SigNoz repo's `deploy/`\n\nfolder, that path has been deprecated in a recent release. The currently supported installer is **Foundry**, a declarative CLI that generates and manages the Compose (or Kubernetes) files for you.\n\nThe actual flow is refreshingly small:\n\n```\n# 1. Install the Foundry CLI\ncurl -fsSL https://signoz.io/foundry.sh | bash\n\n# 2. Declare what you want: Docker Compose, single machine\ncat > casting.yaml << 'EOF'\napiVersion: v1alpha1\nkind: Installation\nmetadata:\n  name: signoz\nspec:\n  deployment:\n    flavor: compose\n    mode: docker\nEOF\n\n# 3. Deploy\nfoundryctl cast -f casting.yaml\n```\n\n`cast`\n\nisn't one opaque command — it chains three stages (`gauge`\n\nvalidates your Docker setup, `forge`\n\nrenders the actual Compose files into a `pours/`\n\ndirectory, `cast`\n\nbrings the stack up), and you can run each stage separately if you want to inspect the generated Compose file before trusting it with `docker compose up`\n\n. That separation is a nice bit of design: I could see exactly what containers were about to start (ClickHouse for storage, a Postgres-backed metastore, a ClickHouse Keeper node, the OTel collector, and the SigNoz UI/API server itself) before committing.\n\nPrerequisite check:Docker Engine 20.10+, the Compose v2 plugin, and at least 4GB of memory allocated to Docker. If containers keep restarting, that memory ceiling is the first thing to raise — SigNoz's docs call this out explicitly, and ClickHouse in particular is memory-hungry on a cold start.\n\nOnce `docker ps`\n\nshows everything healthy, the UI comes up at `http://localhost:8080`\n\n. Ports `4317`\n\n(OTLP gRPC) and `4318`\n\n(OTLP HTTP) are what your application will actually talk to.\n\nAlthough the deployment itself was straightforward, the most useful confirmation came from seeing the services come up cleanly and the SigNoz UI become reachable locally. That moment confirmed the observability stack was ready to receive telemetry from my application.\n\nRather than spinning up a \"hello world\" Flask app just to say I instrumented something, I stood up a stripped-down FastAPI service that mirrors the shape of my real Symptom Intake Agent: it receives a symptom payload, calls out to a downstream \"extraction\" service (standing in for the HPO Extraction Agent), and returns a structured response. Two services, one synchronous hop between them — small enough to reason about, real enough to show actual cross-service tracing instead of a single-span demo.\n\nSigNoz's Python guide recommends `opentelemetry-distro`\n\nfor zero-code auto-instrumentation, which detects your installed libraries and wraps them automatically:\n\n```\npip install opentelemetry-distro opentelemetry-exporter-otlp\nopentelemetry-bootstrap --action=install\n```\n\nThat second command is doing more than it looks like — it scans your installed packages (FastAPI, `requests`\n\n, whatever database driver you're using) and installs the matching OpenTelemetry instrumentation library for each one. Run it *after* your app's dependencies are installed, or it has nothing to detect.\n\nEnvironment variables point the SDK at self-hosted SigNoz instead of the cloud ingestion endpoint:\n\n```\nexport OTEL_RESOURCE_ATTRIBUTES=\"service.name=symptom-intake-agent,service.version=$(git rev-parse --short HEAD)\"\nexport OTEL_EXPORTER_OTLP_ENDPOINT=\"http://localhost:4318\"\nexport OTEL_EXPORTER_OTLP_PROTOCOL=\"http/protobuf\"\nexport OTEL_METRICS_EXPORTER=\"none\"   # traces first, metrics deliberately off for now\n```\n\nCallout — why disable metrics here:`opentelemetry-distro`\n\nenables metrics by default, and the auto-instrumentation for HTTP clients like`requests`\n\nand`httpx`\n\nwill start emitting a duration histogram forevery outgoing callthe moment metrics are on. For a chatty agent pipeline making several LLM and retrieval calls per request, that's a lot of unplanned data volume before you've even decided you want it. Turning it off at first keeps the traces experiment clean; flipping it to`otlp`\n\nlater is a one-line change.\n\nThen the run command changes from `uvicorn main:app`\n\nto:\n\n```\nopentelemetry-instrument uvicorn main:app --host 0.0.0.0 --port 8000\n```\n\n**The gotcha that would have cost me an hour if I hadn't read the docs first:** `opentelemetry-instrument`\n\ndoes not support Uvicorn's `--workers`\n\nflag, and `--reload`\n\noutright breaks instrumentation for *any* framework, not just Uvicorn. The reason is unglamorous but important — the OpenTelemetry SDK's batch span processor runs on a background thread, and Python's fork model doesn't safely carry that thread into forked worker processes unless the server explicitly re-initializes it after fork. Uvicorn's worker mode doesn't do that; Gunicorn (loading the app fresh in each forked worker) does. So for multi-worker production instrumentation, the documented pattern is Gunicorn with Uvicorn workers:\n\n```\nopentelemetry-instrument gunicorn -k uvicorn.workers.UvicornWorker main:app --bind 0.0.0.0:8000\n```\n\nFor a single-worker dev instance, plain `opentelemetry-instrument uvicorn`\n\nwithout `--reload`\n\nis fine, and that's what I used for this warm-up.\n\nOne lesson I took away during setup is that instrumentation deserves the same attention as application code. A small configuration mistake can prevent telemetry from appearing, so validating the OpenTelemetry configuration early saves significant debugging time later.\n\nTo make the tracing story concrete, I sent a single synthetic symptom payload through the two-service slice — a `POST /intake`\n\ncall carrying a small set of HPO-style symptom terms — and watched it end to end instead of looking at aggregate graphs first.\n\nConceptually, the request's journey looks like this:\n\n```\nClient\n  │\n  ▼\nsymptom-intake-agent  (FastAPI, span: POST /intake)\n  │   validates payload\n  │   calls downstream extraction service (span: HTTP POST /extract)\n  ▼\nextraction-service    (FastAPI, span: POST /extract)\n  │   normalizes symptom terms\n  │   returns structured HPO-like terms\n  ▲\nsymptom-intake-agent  (assembles response)\n  │\n  ▼\nClient\n```\n\nBecause both services are auto-instrumented with OpenTelemetry and both export to the same SigNoz collector, the HTTP call between them isn't two disconnected traces — it's one trace with a parent span in `symptom-intake-agent`\n\nand a child span in `extraction-service`\n\n, stitched together by W3C trace-context propagation happening automatically inside the instrumented `requests`\n\n/`httpx`\n\nclient and the receiving FastAPI middleware. That stitching is the entire point of distributed tracing, and it's also the thing that's genuinely hard to bolt on after the fact with hand-rolled logging.\n\nOpening the **Traces** tab in SigNoz and drilling into that request shows a flamegraph / Gantt-style view: the parent span for `POST /intake`\n\nas a wide bar at the top, and the nested `POST /extract`\n\ncall as a shorter bar underneath it, positioned exactly where in time it started and ended relative to the parent.\n\nThis is a small example, but it points at something I wouldn't have appreciated without seeing it laid out visually: **in a chain of agents, latency is additive and attribution is not.** If the whole request takes 900ms and the extraction call alone accounts for 700 of them, that's not a fact you can eyeball from logs with two separate timestamps in two separate files — you'd have to manually subtract wall-clock times and hope your service clocks are synced. In the trace view it's just... the size of the bar.\n\nFor the real twelve-agent pipeline, this is the difference between \"the whole request is slow\" (unhelpful) and \"the RAG Retrieval Engine is 60% of total latency on differential-diagnosis-heavy queries\" (actionable, and exactly the kind of finding that should shape which agent gets optimized first).\n\nWatching the request appear as a complete parent–child trace reinforced the value of distributed tracing. Rather than viewing isolated events, I could understand the entire request lifecycle as one connected execution flow.\n\nA single trace tells you about a single request. Metrics tell you whether that request was typical or an outlier. SigNoz auto-generates APM dashboards from the trace data your services already send — p99 latency, error rate, throughput, and Apdex score per service — without any separate metrics instrumentation required for basic request-level numbers.\n\nThat auto-generation matters for a hackathon timeline: you get a \"is this service healthy\" dashboard for free the moment traces are flowing, before you've written a single custom metric. For anything beyond that — say, tracking token counts per LLM call, or queue depth in an agent's task backlog — you'd add manual metrics instrumentation (counters, histograms) via the OpenTelemetry Python metrics API, which is the natural \"next step\" once the free APM dashboard stops being enough.\n\nThe reason logs still matter even with rich traces: a span tells you *that* something happened in that time window and *how long* it took, but it doesn't automatically carry your application's own log lines — \"symptom set had 3 unrecognized terms,\" say. SigNoz's log pipeline is built to correlate logs with the trace and span ID active at the time the log line was emitted, so from a slow span you can jump to exactly the log lines produced during that span, instead of grepping a log file by approximate timestamp.\n\nFor an agent pipeline where a \"failure\" is often silent — an agent returning an empty or low-confidence result rather than throwing an exception — this correlation is what turns \"the trace shows this span took long\" into \"and here's the log line explaining that it was retrying against a rate-limited LLM endpoint three times before it got a response.\"\n\nTraces and logs answer \"what happened in this one request.\" Dashboards and alerts answer \"should I be watching this at all right now.\" Once telemetry is flowing, the natural next step is a small custom dashboard — p99 latency per agent, error rate per agent, and a panel showing the downstream-call duration breakdown — built with SigNoz's Query Builder, which lets you compose these panels without hand-writing ClickHouse SQL, while still dropping into raw queries if you need something the builder doesn't expose yet.\n\nAlerts close the loop: a rule on error rate or p99 latency crossing a threshold means you find out about a regressing agent from a notification, not from a user complaint or a judge poking at your demo. For a system that's explicitly *not* meant to give numeric diagnostic confidence scores to patients but *is* meant to route them reliably, an alert on the Clinical Triage Engine's error rate is arguably more important than almost any other dashboard panel in the whole project.\n\nAfter exploring dashboards and alerting, it became clear that observability is not only about debugging existing issues but also about identifying performance regressions before they become user-facing problems.\n\nIf I had to choose one capability that stood out during this warm-up, it would be **distributed tracing**. Before exploring SigNoz, I tended to think about debugging as collecting logs from different parts of an application and manually piecing events together. Tracing changes that workflow entirely by presenting a request as a single connected journey across services.\n\nWhat I appreciated most is that tracing provides context rather than isolated events. Instead of simply knowing that a request was slower than expected, I could understand which stage contributed to the overall execution time and how the different services interacted. For applications built from multiple components—especially AI and agentic systems—this end-to-end visibility is far more valuable than reading individual log entries in isolation.\n\nAs I continue building my rare disease navigation platform, distributed tracing is the feature I expect to rely on the most. It gives me confidence that as more agents and external services are added, I will still be able to understand the behaviour of the system without turning debugging into guesswork.\n\nOne thing that initially took time to understand was how logs, metrics, and traces complement rather than replace one another. I first approached them as different views of the same information, but using SigNoz made it clear that each answers a different question: metrics reveal trends, logs provide detailed events, and traces explain the lifecycle of an individual request.\n\nThe biggest insight for me was recognizing that observability is something to build into an application from the beginning rather than adding only when problems appear. Once telemetry is available, understanding system behaviour becomes significantly easier because every request carries useful operational context alongside the application logic.\n\nIf I could give my past self one piece of advice, it would be simple: treat observability as part of the development process, not as a finishing step. That mindset will be even more valuable as I build the complete AI-agent workflow for the hackathon.\n\n`--reload`\n\nare real and documented for a reason — hit them once, and you'll remember the fix forever.My project's whole premise is routing patients through a system they can't see — different agents, different confidence signals, a government CoE network behind it all. It would be a strange kind of hypocrisy to build that system and then have zero visibility into how *my own* pipeline behaves in production. This warm-up convinced me that wiring up SigNoz from day one of the real build — not as an afterthought before a demo — is the right call, because the moment I add agent nine through twelve, tracing is the only thing that will keep \"which component broke\" from turning back into archaeology.\n\nIf you're building anything with more than one moving part for this hackathon — agents calling agents, services calling services — do the same warm-up before your sprint clock starts. It's a few hours now against a much worse debugging session later.\n\n*Built for the WeMakeDevs × SigNoz \"Agents of SigNoz\" hackathon warm-up blog challenge. Self-hosted SigNoz via Foundry, OpenTelemetry auto-instrumentation on FastAPI, submitted ahead of the July 19, 2026 deadline.*", "url": "https://wpnews.pro/news/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz", "canonical_source": "https://dev.to/vezzu_ruthvik/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz-2l1f", "published_at": "2026-07-10 13:45:41+00:00", "updated_at": "2026-07-10 14:15:34.743334+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["SigNoz", "LangGraph", "WeMakeDevs", "Foundry"], "alternates": {"html": "https://wpnews.pro/news/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz", "markdown": "https://wpnews.pro/news/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz.md", "text": "https://wpnews.pro/news/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz.txt", "jsonld": "https://wpnews.pro/news/i-gave-my-ai-agent-pipeline-a-nervous-system-with-signoz.jsonld"}}