# AgentATC > Source: > Published: 2026-07-25 22:54:59+00:00 Track 3: AI & Agent Observability — "Agents of SigNoz" Hackathon (WeMakeDevs × SigNoz, July 2026) Every team at a hackathon like this is going to show you an agent that's slow, or an agent that errors. Tasks bounce endlessly between an Executor and a Critic. Work is quietly repeated even after it's completed. Sometimes a task simply sits there because no agent picks it up. Meanwhile, LLM tokens continue to burn and costs keep increasing. Traditional observability focuses on individual agent calls: It was never built to observe the thing that actually breaks multi-agent systems: Coordination between agents. A task endlessly bouncing between two agents doesn't look like a failure. To most observability tools, everything still looks perfectly healthy. AgentATC is a real (not scripted) three-agent workflow consisting of a Planner, an Executor, and a Critic that collaborates to generate a competitive analysis report. **The workflow is straightforward:** That workflow alone isn't what makes AgentATC interesting. What makes it different is that every hand-off between agents becomes a first-class OpenTelemetry span. Instead of trying to infer coordination after the fact from timestamps, AgentATC records coordination directly. Every span such as: carries information about: Which agent initiated the action Which agent received it Why the hand-off happened Coordination becomes observable data rather than an invisible gap between API calls. Every task is assigned: One task_id One root span (task.process) One connected trace throughout the execution This architectural decision is what keeps the entire workflow connected. Without it, every HTTP request between the Orchestrator, Planner, Executor, and Critic would generate independent traces, fragmenting the execution into unrelated pieces. Each subtask runs inside its own nested span while the Executor and Critic iterate until convergence. Whenever the Critic rejects a draft, agentatc_handoffs_total{reason="reject"} is incremented. When a task stalls, nothing throws an exception. Nothing crashes. Instead, agentatc_task_last_update_timestamp simply stops moving forward, providing the only indication that no agent is making progress anymore. **A Successful Execution** Submit the goal: "Write a competitive analysis of three observability vendors: SigNoz, Datadog, and New Relic." The Planner creates three subtasks. Two of them— SigNoz New Relic —converge immediately and pass the Critic on the first review. That's the expected path. It should look boring. Successful coordination shouldn't generate excitement. When Things Go Wrong The Datadog subtask is intentionally different. Its mock search tool can never retrieve pricing information, while the Critic's evaluation rubric requires pricing to approve the draft. As a result, the Executor can never satisfy the Critic. Every execution guarantees: Five review iterations Five rejections One escalation No randomness. No hoping an LLM behaves the same during a live demo. Because everything shares one trace ID, opening the trace reveals the entire history as one connected execution, showing every Executor ↔ Critic hand-off across all services. Complete Transparency with SigNoz Dashboards and Copilot The Swarm Health Overview dashboard tracks: The Thrash Leaderboard ranks subtasks by rejection count. Datadog consistently appears at the top, making the coordination failure immediately visible. Two production alerts help detect coordination failures early: Task Thrashing Task Stalled Both alerts can automatically trigger the Copilot. The Copilot uses SigNoz's MCP Server to investigate another agent's failure the way an on-call engineer would. It retrieves telemetry using: Most importantly, it is built to say "No data found" instead of fabricating an answer. **During one escalated execution it concluded:** "The Critic consistently rejected the submissions for the same reason: missing pricing information... this indicates a TOOLING GAP. The executor's data sources do not provide the required pricing information, leading to consistent rejections by the critic." That distinction is the entire value proposition. Not simply that a task failed five times. But why it failed, backed by actual telemetry. We didn't bolt dashboards onto the project after the fact. Traces, metrics, and logs all flow through SigNoz for real analytical work. Coordination metrics are computed directly from raw telemetry in SigNoz's query layer instead of being pre-calculated inside application code. While migrating dashboards to the Query Builder, we discovered that one PromQL query grouped results using a label that didn't actually exist. The dashboard rendered successfully. The data was wrong. That wasn't a SigNoz issue. It was an instrumentation bug in our own project that only became visible because we validated the dashboards against live telemetry instead of assuming a rendered graph must be correct. The failure mode we set out to expose includes: These failures produce almost no useful signal in traditional monitoring systems. Building the metrics to observe them forced us to think about coordination as telemetry, rather than treating it as a side effect of healthy service calls. Going back to validate our dashboards against live data instead of trusting that they rendered correctly also uncovered a genuine instrumentation bug that would otherwise have shipped unnoticed. We didn't build another AI agent demo. We built the instrument panel for the failure mode every multi-agent team is about to discover the hard way. The one where: Everything looks healthy. Nothing ever fails. And the system is stuck anyway.