{"slug": "building-an-incident-triage-agent-with-full-observability-in-signoz", "title": "Building an Incident Triage Agent with Full Observability in SigNoz", "summary": "A developer built an Incident Triage Agent for the Agents of SigNoz hackathon that makes AI agent decision-making fully observable using OpenTelemetry spans. The agent simulates an on-call engineer's assistant during production incidents, with each step wrapped in its own span so a single investigation appears as one connected trace in SigNoz. The developer found that essentially all latency came from LLM calls, not tool calls, highlighting the importance of agent observability.", "body_md": "AI agents chain LLM calls and tool calls together to make decisions. But\n\nwhen something goes wrong — a slow response, an unexpected answer, a\n\nrunaway cost — you're stuck guessing. You can't debug what you can't see.\n\nFor the \"Agents of SigNoz\" hackathon, I wanted to build something that\n\naddresses this directly: an AI agent whose entire decision-making process\n\nis fully observable, end to end.\n\nAn **Incident Triage Agent** — a small AI agent that simulates what an\n\non-call engineer's assistant might do during a production incident:\n\nEvery one of these five steps is wrapped in its own OpenTelemetry span, so\n\na single incident investigation shows up in SigNoz as one connected trace\n\nwith all five child spans nested underneath it.\n\n`gemini-3.5-flash`\n\n) for the two LLM callsEach span carries custom attributes beyond the defaults — token counts,\n\nlatency in seconds, an estimated cost, the severity level the agent\n\nassigned, and even a simple heuristic flag for potential hallucination\n\n(whether the final summary references something that was never in the\n\nlogs or runbook). Opening a trace in SigNoz's flame graph view immediately\n\nshows where the time went — in my case, the two LLM calls consistently\n\ntook 10-15 seconds each, while all three tool calls combined took under a\n\nsecond. That's not obvious from reading code; it's obvious from a trace.\n\nI built two dashboards directly from trace data using SigNoz's query\n\nbuilder:\n\nI set up a threshold alert on the planning LLM call: if `agent.llm.plan`\n\ntakes longer than 15 seconds on average, it fires. This is the kind of\n\nthing you'd actually want in production — a way to know your agent is\n\ndegrading before someone notices the user-facing symptom.\n\nThe most interesting finding, honestly, wasn't about SigNoz — it was\n\nabout my own agent. Once I could see the trace breakdown, it became\n\nobvious that essentially all the latency in this pipeline comes from the\n\ntwo LLM calls, not the tool calls. That's an unglamorous but important\n\ninsight: if I wanted to optimize this agent, the tool calls are not\n\nwhere I'd spend my time.\n\nThat's the whole pitch for agent observability — it turns \"the agent\n\nfelt slow\" into \"the agent's planning step specifically took 12 seconds,\n\nhere's the trace, here's the token count, here's the cost.\" You can't\n\nhave that conversation without instrumentation.\n\nFull code, setup instructions, and the Foundry deployment config are on\n\nGitHub: [github.com/aadvik93/incident-triage-agent](https://github.com/aadvik93/incident-triage-agent)\n\n*Built for the Agents of SigNoz\nhackathon, Track 01: AI & Agent Observability.*", "url": "https://wpnews.pro/news/building-an-incident-triage-agent-with-full-observability-in-signoz", "canonical_source": "https://dev.to/aadvik_krishna_2d1baa4c0a/building-an-incident-triage-agent-with-full-observability-in-signoz-55b0", "published_at": "2026-07-24 01:00:59+00:00", "updated_at": "2026-07-24 01:33:16.358546+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "large-language-models"], "entities": ["SigNoz", "OpenTelemetry", "Gemini 3.5 Flash", "GitHub", "Agents of SigNoz hackathon"], "alternates": {"html": "https://wpnews.pro/news/building-an-incident-triage-agent-with-full-observability-in-signoz", "markdown": "https://wpnews.pro/news/building-an-incident-triage-agent-with-full-observability-in-signoz.md", "text": "https://wpnews.pro/news/building-an-incident-triage-agent-with-full-observability-in-signoz.txt", "jsonld": "https://wpnews.pro/news/building-an-incident-triage-agent-with-full-observability-in-signoz.jsonld"}}