The Dashboard Said My AI Team Finished. One Agent Never Even Started.

A developer using SigNoz discovered that their AI agent workflow silently completed despite a research agent timing out and returning no data. The workflow's fallback logic caught the timeout and passed an empty context to subsequent agents, producing a polished but incomplete final output. OpenTelemetry tracing in SigNoz revealed the hidden failure by showing zero documents returned, fallback usage, and a low review score.

My AI workflow reported that the task was complete. The API returned 200 OK . The user interface showed every agent as completed. A final document was generated and delivered successfully. There was only one problem. The research agent had never returned any research. The workflow silently caught a timeout, passed an empty response to the writing agent, and continued as though nothing had happened. The final output looked confident and polished, but it was based on incomplete context. This was not a normal application crash. There was no obvious error page, failed request, or red status indicator. From the outside, everything looked healthy. SigNoz helped me see what had actually happened inside the workflow. The application uses several AI agents to complete one project: The expected workflow is: User Request ↓ Lead Agent ↓ Research Agent ↓ Writing Agent ↓ Review Agent ↓ Delivery Agent Each agent depends on the result of the previous agent. If the research agent fails, the writing agent should not continue as though valid research was available. However, my initial implementation used fallback logic that allowed the workflow to continue. try { researchContext = await runResearchAgent task ; } catch error { console.error "Research agent failed", error ; researchContext = ; } This prevented the entire request from crashing, but it created a more dangerous problem. The system returned a technically successful response even though an essential part of the workflow had failed. I instrumented the workflow using OpenTelemetry and sent the telemetry data to SigNoz. I created one parent span for the complete project run: project.run Each agent execution became a child span: lead.plan researcher.search writer.draft reviewer.validate delivery.export I also attached attributes that describe what happened during each step. span.setAttributes { "agent.name": "researcher", "task.id": taskId, "handoff.from": "lead", "handoff.to": "researcher", "retry.count": retryCount, "documents.returned": documents.length, "fallback.used": fallbackUsed, } ; For the writing and review agents, I tracked additional values: span.setAttributes { "tokens.input": inputTokens, "tokens.output": outputTokens, "review.score": reviewScore, "output.approved": outputApproved, } ; These attributes allowed me to investigate more than basic application uptime. I could now see: To test the workflow, I deliberately introduced a timeout into the research service. The research agent was configured to stop waiting after TIMEOUT VALUE seconds. js const researchContext = await Promise.race runResearchAgent task , timeoutAfter TIMEOUT VALUE , ; The research service did not respond within the allowed time. The application caught the exception and continued with an empty research context. The final API response still looked successful: { "status": "completed", "httpStatus": 200, "documentGenerated": true } However, the trace attributes told a different story: workflow.status: completed researcher.documents.returned: 0 researcher.retry.count: REAL RETRY COUNT fallback.used: true review.score: REAL REVIEW SCORE The workflow was technically completed but functionally degraded. Inside SigNoz, the complete workflow appeared as one distributed trace. The trace waterfall made the problem immediately visible. project.run REAL TOTAL TIME ├── lead.plan REAL TIME ├── researcher.search REAL TIME — timeout ├── writer.draft REAL TIME — fallback context used ├── reviewer.validate REAL TIME — low review score └── delivery.export REAL TIME The research span consumed most of the workflow time before timing out. The writing agent then started with no research documents, but the span itself was still marked as successful because the fallback logic prevented an exception from propagating. This showed me an important limitation in my original monitoring logic: A successful span did not necessarily mean that the agent completed its intended responsibility. The system needed to track semantic success, not only technical success. The trace showed me where the delay occurred. The correlated logs explained why the workflow continued. The research span contained a log similar to: Research provider timed out after REAL TIMEOUT seconds. Continuing with empty research context. Another log from the writing agent showed: Writing agent started with 0 research documents. Fallback context enabled. Without trace and log correlation, these messages would have been difficult to connect. A timeout log by itself does not explain which user request was affected. A completed API request does not explain that an upstream agent failed. SigNoz connected the logs to the exact trace, task, and agent span. I created a dashboard focused on AI-agent workflow health rather than only infrastructure health. The dashboard included: This panel shows how long each agent takes to complete its work. It makes unusually slow agents and timeout patterns easier to identify. This tracks handoffs where: handoff.status = "success" or: fallback.used = true This shows whether one agent is consuming an unusually large portion of the workflow’s token budget. This was the most important panel. A workflow should not be considered fully successful when it completed using missing or fallback context. Traditional alerts often focus on conditions such as: HTTP status = 500 That would not detect this problem because the API returned 200 OK . Instead, I created an alert for workflows that completed while using fallback data. workflow.status = "completed" AND fallback.used = true I named the alert: Successful run with degraded agent handoff This alert captures a category of failure that normal uptime monitoring misses. The infrastructure can be available, the API can return successfully, and the user can receive an output while the workflow is still logically incorrect. The original logic treated missing research as an acceptable fallback. if researchFailed { researchContext = ; continueWorkflow ; } I changed it so that an essential agent failure marks the complete workflow as degraded. if researchFailed || researchDocuments.length === 0 { span.setAttributes { "workflow.status": "degraded", "handoff.status": "failed", "fallback.used": false, } ; throw new Error "Research agent failed. Drafting stopped to prevent unsupported output." ; } The revised behavior is now: Research agent fails ↓ Workflow marked as degraded ↓ Writing agent does not start ↓ User receives a clear retry message I ran the same test again. This time, SigNoz showed that the workflow stopped at the research span instead of producing an unsupported final document. The application no longer confused technical completion with valid completion. Before adding observability, I could see whether the application returned a response. After adding SigNoz, I could see whether each agent actually fulfilled its responsibility. That distinction matters in AI systems. A normal application often fails through an exception, unavailable service, or unsuccessful HTTP request. An AI workflow can fail more quietly: The most dangerous AI failures are not always crashes. Sometimes every technical component appears healthy while the final result is still wrong. SigNoz helped me observe the complete chain of decisions, handoffs, retries, logs, and outputs behind one AI-generated result. The dashboard originally told me that my AI team had completed the task. The trace showed me the truth: one of the most important agents had never completed its work.