Building an Incident Triage Agent with Full Observability in SigNoz

A developer built an Incident Triage Agent for the Agents of SigNoz hackathon that makes AI agent decision-making fully observable using OpenTelemetry spans. The agent simulates an on-call engineer's assistant during production incidents, with each step wrapped in its own span so a single investigation appears as one connected trace in SigNoz. The developer found that essentially all latency came from LLM calls, not tool calls, highlighting the importance of agent observability.

AI agents chain LLM calls and tool calls together to make decisions. But when something goes wrong — a slow response, an unexpected answer, a runaway cost — you're stuck guessing. You can't debug what you can't see. For the "Agents of SigNoz" hackathon, I wanted to build something that addresses this directly: an AI agent whose entire decision-making process is fully observable, end to end. An Incident Triage Agent — a small AI agent that simulates what an on-call engineer's assistant might do during a production incident: Every one of these five steps is wrapped in its own OpenTelemetry span, so a single incident investigation shows up in SigNoz as one connected trace with all five child spans nested underneath it. gemini-3.5-flash for the two LLM callsEach span carries custom attributes beyond the defaults — token counts, latency in seconds, an estimated cost, the severity level the agent assigned, and even a simple heuristic flag for potential hallucination whether the final summary references something that was never in the logs or runbook . Opening a trace in SigNoz's flame graph view immediately shows where the time went — in my case, the two LLM calls consistently took 10-15 seconds each, while all three tool calls combined took under a second. That's not obvious from reading code; it's obvious from a trace. I built two dashboards directly from trace data using SigNoz's query builder: I set up a threshold alert on the planning LLM call: if agent.llm.plan takes longer than 15 seconds on average, it fires. This is the kind of thing you'd actually want in production — a way to know your agent is degrading before someone notices the user-facing symptom. The most interesting finding, honestly, wasn't about SigNoz — it was about my own agent. Once I could see the trace breakdown, it became obvious that essentially all the latency in this pipeline comes from the two LLM calls, not the tool calls. That's an unglamorous but important insight: if I wanted to optimize this agent, the tool calls are not where I'd spend my time. That's the whole pitch for agent observability — it turns "the agent felt slow" into "the agent's planning step specifically took 12 seconds, here's the trace, here's the token count, here's the cost." You can't have that conversation without instrumentation. Full code, setup instructions, and the Foundry deployment config are on GitHub: github.com/aadvik93/incident-triage-agent https://github.com/aadvik93/incident-triage-agent Built for the Agents of SigNoz hackathon, Track 01: AI & Agent Observability.