{"slug": "agentatc", "title": "AgentATC", "summary": "AgentATC is a multi-agent workflow that records every hand-off between agents as a first-class OpenTelemetry span, making coordination failures visible. The system exposes metrics like hand-off rejection counts and task staleness, and uses SigNoz dashboards and a Copilot to detect issues such as task thrashing and stalled tasks.", "body_md": "Track 3: AI & Agent Observability — \"Agents of SigNoz\" Hackathon (WeMakeDevs × SigNoz, July 2026)\n\nEvery team at a hackathon like this is going to show you an agent that's slow, or an agent that errors.\n\nTasks bounce endlessly between an Executor and a Critic. Work is quietly repeated even after it's completed. Sometimes a task simply sits there because no agent picks it up. Meanwhile, LLM tokens continue to burn and costs keep increasing.\n\nTraditional observability focuses on individual agent calls:\n\nIt was never built to observe the thing that actually breaks multi-agent systems:\n\nCoordination between agents.\n\nA task endlessly bouncing between two agents doesn't look like a failure.\n\nTo most observability tools, everything still looks perfectly healthy.\n\nAgentATC is a real (not scripted) three-agent workflow consisting of a Planner, an Executor, and a Critic that collaborates to generate a competitive analysis report.\n\n**The workflow is straightforward:**\n\nThat workflow alone isn't what makes AgentATC interesting.\n\nWhat makes it different is that every hand-off between agents becomes a first-class OpenTelemetry span.\n\nInstead of trying to infer coordination after the fact from timestamps, AgentATC records coordination directly.\n\nEvery span such as:\n\ncarries information about:\n\nWhich agent initiated the action\n\nWhich agent received it\n\nWhy the hand-off happened\n\nCoordination becomes observable data rather than an invisible gap between API calls.\n\nEvery task is assigned:\n\nOne task_id\n\nOne root span (task.process)\n\nOne connected trace throughout the execution\n\nThis architectural decision is what keeps the entire workflow connected.\n\nWithout it, every HTTP request between the Orchestrator, Planner, Executor, and Critic would generate independent traces, fragmenting the execution into unrelated pieces.\n\nEach subtask runs inside its own nested span while the Executor and Critic iterate until convergence.\n\nWhenever the Critic rejects a draft,\n\nagentatc_handoffs_total{reason=\"reject\"}\n\nis incremented.\n\nWhen a task stalls, nothing throws an exception.\n\nNothing crashes.\n\nInstead,\n\nagentatc_task_last_update_timestamp\n\nsimply stops moving forward, providing the only indication that no agent is making progress anymore.\n\n**A Successful Execution**\n\nSubmit the goal:\n\n\"Write a competitive analysis of three observability vendors: SigNoz, Datadog, and New Relic.\"\n\nThe Planner creates three subtasks.\n\nTwo of them—\n\nSigNoz\n\nNew Relic\n\n—converge immediately and pass the Critic on the first review.\n\nThat's the expected path.\n\nIt should look boring.\n\nSuccessful coordination shouldn't generate excitement.\n\nWhen Things Go Wrong\n\nThe Datadog subtask is intentionally different.\n\nIts mock search tool can never retrieve pricing information, while the Critic's evaluation rubric requires pricing to approve the draft.\n\nAs a result, the Executor can never satisfy the Critic.\n\nEvery execution guarantees:\n\nFive review iterations\n\nFive rejections\n\nOne escalation\n\nNo randomness.\n\nNo hoping an LLM behaves the same during a live demo.\n\nBecause everything shares one trace ID, opening the trace reveals the entire history as one connected execution, showing every Executor ↔ Critic hand-off across all services.\n\nComplete Transparency with SigNoz Dashboards and Copilot\n\nThe Swarm Health Overview dashboard tracks:\n\nThe Thrash Leaderboard ranks subtasks by rejection count.\n\nDatadog consistently appears at the top, making the coordination failure immediately visible.\n\nTwo production alerts help detect coordination failures early:\n\nTask Thrashing\n\nTask Stalled\n\nBoth alerts can automatically trigger the Copilot.\n\nThe Copilot uses SigNoz's MCP Server to investigate another agent's failure the way an on-call engineer would.\n\nIt retrieves telemetry using:\n\nMost importantly, it is built to say \"No data found\" instead of fabricating an answer.\n\n**During one escalated execution it concluded:**\n\n\"The Critic consistently rejected the submissions for the same reason: missing pricing information... this indicates a TOOLING GAP. The executor's data sources do not provide the required pricing information, leading to consistent rejections by the critic.\"\n\nThat distinction is the entire value proposition.\n\nNot simply that a task failed five times.\n\nBut why it failed, backed by actual telemetry.\n\nWe didn't bolt dashboards onto the project after the fact.\n\nTraces, metrics, and logs all flow through SigNoz for real analytical work.\n\nCoordination metrics are computed directly from raw telemetry in SigNoz's query layer instead of being pre-calculated inside application code.\n\nWhile migrating dashboards to the Query Builder, we discovered that one PromQL query grouped results using a label that didn't actually exist.\n\nThe dashboard rendered successfully.\n\nThe data was wrong.\n\nThat wasn't a SigNoz issue.\n\nIt was an instrumentation bug in our own project that only became visible because we validated the dashboards against live telemetry instead of assuming a rendered graph must be correct.\n\nThe failure mode we set out to expose includes:\n\nThese failures produce almost no useful signal in traditional monitoring systems.\n\nBuilding the metrics to observe them forced us to think about coordination as telemetry, rather than treating it as a side effect of healthy service calls.\n\nGoing back to validate our dashboards against live data instead of trusting that they rendered correctly also uncovered a genuine instrumentation bug that would otherwise have shipped unnoticed.\n\nWe didn't build another AI agent demo.\n\nWe built the instrument panel for the failure mode every multi-agent team is about to discover the hard way.\n\nThe one where:\n\nEverything looks healthy.\n\nNothing ever fails.\n\nAnd the system is stuck anyway.", "url": "https://wpnews.pro/news/agentatc", "canonical_source": "https://dev.to/zephyr_77/agentatc-93g", "published_at": "2026-07-25 22:54:59+00:00", "updated_at": "2026-07-25 23:31:21.223875+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools"], "entities": ["SigNoz", "Datadog", "New Relic", "WeMakeDevs", "AgentATC", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/agentatc", "markdown": "https://wpnews.pro/news/agentatc.md", "text": "https://wpnews.pro/news/agentatc.txt", "jsonld": "https://wpnews.pro/news/agentatc.jsonld"}}