{"slug": "the-dashboard-said-my-ai-team-finished-one-agent-never-even-started", "title": "The Dashboard Said My AI Team Finished. One Agent Never Even Started.", "summary": "A developer using SigNoz discovered that their AI agent workflow silently completed despite a research agent timing out and returning no data. The workflow's fallback logic caught the timeout and passed an empty context to subsequent agents, producing a polished but incomplete final output. OpenTelemetry tracing in SigNoz revealed the hidden failure by showing zero documents returned, fallback usage, and a low review score.", "body_md": "My AI workflow reported that the task was complete.\n\nThe API returned `200 OK`\n\n. The user interface showed every agent as completed. A final document was generated and delivered successfully.\n\nThere was only one problem.\n\nThe research agent had never returned any research.\n\nThe workflow silently caught a timeout, passed an empty response to the writing agent, and continued as though nothing had happened. The final output looked confident and polished, but it was based on incomplete context.\n\nThis was not a normal application crash. There was no obvious error page, failed request, or red status indicator.\n\nFrom the outside, everything looked healthy.\n\nSigNoz helped me see what had actually happened inside the workflow.\n\nThe application uses several AI agents to complete one project:\n\nThe expected workflow is:\n\n```\nUser Request\n    ↓\nLead Agent\n    ↓\nResearch Agent\n    ↓\nWriting Agent\n    ↓\nReview Agent\n    ↓\nDelivery Agent\n```\n\nEach agent depends on the result of the previous agent.\n\nIf the research agent fails, the writing agent should not continue as though valid research was available. However, my initial implementation used fallback logic that allowed the workflow to continue.\n\n```\ntry {\n  researchContext = await runResearchAgent(task);\n} catch (error) {\n  console.error(\"Research agent failed\", error);\n  researchContext = [];\n}\n```\n\nThis prevented the entire request from crashing, but it created a more dangerous problem.\n\nThe system returned a technically successful response even though an essential part of the workflow had failed.\n\nI instrumented the workflow using OpenTelemetry and sent the telemetry data to SigNoz.\n\nI created one parent span for the complete project run:\n\n```\nproject.run\n```\n\nEach agent execution became a child span:\n\n```\nlead.plan\nresearcher.search\nwriter.draft\nreviewer.validate\ndelivery.export\n```\n\nI also attached attributes that describe what happened during each step.\n\n```\nspan.setAttributes({\n  \"agent.name\": \"researcher\",\n  \"task.id\": taskId,\n  \"handoff.from\": \"lead\",\n  \"handoff.to\": \"researcher\",\n  \"retry.count\": retryCount,\n  \"documents.returned\": documents.length,\n  \"fallback.used\": fallbackUsed,\n});\n```\n\nFor the writing and review agents, I tracked additional values:\n\n```\nspan.setAttributes({\n  \"tokens.input\": inputTokens,\n  \"tokens.output\": outputTokens,\n  \"review.score\": reviewScore,\n  \"output.approved\": outputApproved,\n});\n```\n\nThese attributes allowed me to investigate more than basic application uptime.\n\nI could now see:\n\nTo test the workflow, I deliberately introduced a timeout into the research service.\n\nThe research agent was configured to stop waiting after `[TIMEOUT VALUE]`\n\nseconds.\n\n``` js\nconst researchContext = await Promise.race([\n  runResearchAgent(task),\n  timeoutAfter([TIMEOUT VALUE]),\n]);\n```\n\nThe research service did not respond within the allowed time.\n\nThe application caught the exception and continued with an empty research context.\n\nThe final API response still looked successful:\n\n```\n{\n  \"status\": \"completed\",\n  \"httpStatus\": 200,\n  \"documentGenerated\": true\n}\n```\n\nHowever, the trace attributes told a different story:\n\n```\nworkflow.status: completed\nresearcher.documents.returned: 0\nresearcher.retry.count: [REAL RETRY COUNT]\nfallback.used: true\nreview.score: [REAL REVIEW SCORE]\n```\n\nThe workflow was technically completed but functionally degraded.\n\nInside SigNoz, the complete workflow appeared as one distributed trace.\n\nThe trace waterfall made the problem immediately visible.\n\n```\nproject.run             [REAL TOTAL TIME]\n├── lead.plan           [REAL TIME]\n├── researcher.search   [REAL TIME] — timeout\n├── writer.draft        [REAL TIME] — fallback context used\n├── reviewer.validate   [REAL TIME] — low review score\n└── delivery.export     [REAL TIME]\n```\n\nThe research span consumed most of the workflow time before timing out.\n\nThe writing agent then started with no research documents, but the span itself was still marked as successful because the fallback logic prevented an exception from propagating.\n\nThis showed me an important limitation in my original monitoring logic:\n\nA successful span did not necessarily mean that the agent completed its intended responsibility.\n\nThe system needed to track semantic success, not only technical success.\n\nThe trace showed me where the delay occurred. The correlated logs explained why the workflow continued.\n\nThe research span contained a log similar to:\n\n```\nResearch provider timed out after [REAL TIMEOUT] seconds.\nContinuing with empty research context.\n```\n\nAnother log from the writing agent showed:\n\n```\nWriting agent started with 0 research documents.\nFallback context enabled.\n```\n\nWithout trace and log correlation, these messages would have been difficult to connect.\n\nA timeout log by itself does not explain which user request was affected. A completed API request does not explain that an upstream agent failed.\n\nSigNoz connected the logs to the exact trace, task, and agent span.\n\nI created a dashboard focused on AI-agent workflow health rather than only infrastructure health.\n\nThe dashboard included:\n\nThis panel shows how long each agent takes to complete its work.\n\nIt makes unusually slow agents and timeout patterns easier to identify.\n\nThis tracks handoffs where:\n\n```\nhandoff.status != \"success\"\n```\n\nor:\n\n```\nfallback.used = true\n```\n\nThis shows whether one agent is consuming an unusually large portion of the workflow’s token budget.\n\nThis was the most important panel.\n\nA workflow should not be considered fully successful when it completed using missing or fallback context.\n\nTraditional alerts often focus on conditions such as:\n\n```\nHTTP status >= 500\n```\n\nThat would not detect this problem because the API returned `200 OK`\n\n.\n\nInstead, I created an alert for workflows that completed while using fallback data.\n\n```\nworkflow.status = \"completed\"\nAND fallback.used = true\n```\n\nI named the alert:\n\n```\nSuccessful run with degraded agent handoff\n```\n\nThis alert captures a category of failure that normal uptime monitoring misses.\n\nThe infrastructure can be available, the API can return successfully, and the user can receive an output while the workflow is still logically incorrect.\n\nThe original logic treated missing research as an acceptable fallback.\n\n```\nif (researchFailed) {\n  researchContext = [];\n  continueWorkflow();\n}\n```\n\nI changed it so that an essential agent failure marks the complete workflow as degraded.\n\n```\nif (researchFailed || researchDocuments.length === 0) {\n  span.setAttributes({\n    \"workflow.status\": \"degraded\",\n    \"handoff.status\": \"failed\",\n    \"fallback.used\": false,\n  });\n\n  throw new Error(\n    \"Research agent failed. Drafting stopped to prevent unsupported output.\"\n  );\n}\n```\n\nThe revised behavior is now:\n\n```\nResearch agent fails\n        ↓\nWorkflow marked as degraded\n        ↓\nWriting agent does not start\n        ↓\nUser receives a clear retry message\n```\n\nI ran the same test again.\n\nThis time, SigNoz showed that the workflow stopped at the research span instead of producing an unsupported final document.\n\nThe application no longer confused technical completion with valid completion.\n\nBefore adding observability, I could see whether the application returned a response.\n\nAfter adding SigNoz, I could see whether each agent actually fulfilled its responsibility.\n\nThat distinction matters in AI systems.\n\nA normal application often fails through an exception, unavailable service, or unsuccessful HTTP request.\n\nAn AI workflow can fail more quietly:\n\nThe most dangerous AI failures are not always crashes.\n\nSometimes every technical component appears healthy while the final result is still wrong.\n\nSigNoz helped me observe the complete chain of decisions, handoffs, retries, logs, and outputs behind one AI-generated result.\n\nThe dashboard originally told me that my AI team had completed the task.\n\nThe trace showed me the truth: one of the most important agents had never completed its work.", "url": "https://wpnews.pro/news/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started", "canonical_source": "https://dev.to/bilalferoz/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started-1e0l", "published_at": "2026-07-18 20:48:20+00:00", "updated_at": "2026-07-18 21:27:51.760345+00:00", "lang": "en", "topics": ["ai-agents", "ai-infrastructure", "developer-tools", "ai-safety"], "entities": ["SigNoz", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started", "markdown": "https://wpnews.pro/news/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started.md", "text": "https://wpnews.pro/news/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started.txt", "jsonld": "https://wpnews.pro/news/the-dashboard-said-my-ai-team-finished-one-agent-never-even-started.jsonld"}}