I Built an AI Team That Investigates Production Incidents Better Than Most Engineers

A developer built TattvaAI, an AI-powered incident investigation platform that uses six specialized agents to autonomously analyze production incidents. The system, built for the SigNoz 'Agents of SigNoz' Hackathon, correlates traces, logs, metrics, and dependencies to identify root causes faster than manual investigation.

Built for the SigNoz "Agents of SigNoz" Hackathon — July 2026 Every engineer who has been on call knows the feeling. An alert fires in the middle of the night. Dashboards turn red. Error rates spike. Customers begin reporting failures. The incident itself may eventually be simple—a database timeout, an exhausted Redis connection pool, a failing downstream dependency—but discovering the actual root cause is often the hardest part. The investigation usually looks something like this: By the time the root cause is identified, hours may have passed. In many organizations, the fix itself takes only a few minutes. The investigation is the real bottleneck. That observation became the motivation behind TattvaAI . TattvaAI is an AI-powered incident investigation platform that automatically performs the work experienced Site Reliability Engineers SREs do during production incidents. Instead of presenting engineers with dashboards full of raw telemetry, TattvaAI investigates the incident autonomously, correlates evidence from multiple observability signals, identifies the most probable root cause, and explains its reasoning. Rather than replacing engineers, it removes the repetitive investigative work so engineers can focus on making decisions and restoring production faster. The entire system was built for the SigNoz "Agents of SigNoz" Hackathon using SigNoz as the observability platform and LangGraph to orchestrate multiple specialized AI agents. Today's observability platforms have become extremely good at collecting telemetry. They provide: The data exists. The problem is interpretation. When production fails, engineers still have to manually connect all these signals together. A single incident often requires answering questions like: Answering these questions manually consumes most of the incident response timeline. The challenge isn't a lack of observability. It's a lack of automated reasoning. Instead of creating one large AI assistant, I designed six independent AI agents , each responsible for one domain of observability. Every agent specializes in gathering evidence. Later, a reasoning engine combines that evidence into a complete investigation. Responsible for analyzing distributed traces. It identifies: Instead of simply reporting slow requests, it determines where latency originates inside an entire request path. Responsible for log intelligence. It searches for: Rather than reading thousands of log entries manually, the agent extracts only the evidence relevant to the incident. Responsible for infrastructure and application health. It continuously evaluates: The goal is identifying abnormal behavior that explains observed failures. Distributed systems rarely fail in isolation. This agent maps: Instead of asking "Which service is broken?" , it answers: "Which service caused every other service to fail?" Modern monitoring systems often generate dozens of alerts during a single outage. Many are secondary effects. This agent correlates alerts together and removes noise by identifying which alerts are symptoms versus actual causes. Production systems repeat themselves. This agent searches previous incidents and historical telemetry to answer questions such as: Instead of starting every investigation from scratch, TattvaAI learns from operational history. Each AI agent works independently. Their findings become structured evidence rather than isolated observations. LangGraph orchestrates the entire workflow by: This architecture makes investigations deterministic, explainable, and extensible. Adding another specialist agent later becomes straightforward. One of the biggest design decisions was integrating directly with SigNoz . Instead of treating SigNoz as another dashboard, TattvaAI accesses telemetry through the Model Context Protocol MCP . This allows agents to retrieve: without manually navigating dashboards. SigNoz becomes the source of truth, while TattvaAI becomes the reasoning layer built on top of it. This separation keeps observability and intelligence cleanly decoupled. To evaluate the platform, I created a realistic microservices environment with multiple containerized services generating live telemetry. One investigation looked like this. An e-commerce checkout system suddenly experienced an 18% error rate . An engineer would typically: Total investigation time: 2–3 hours Within seconds the AI agents produced the following evidence. Trace Agent Payment service latency increased to 4800 ms, over four times the normal baseline. Logs Agent Detected 143 Redis timeout exceptions originating from the payment service. Metrics Agent Redis CPU reached 98%, and connection pool utilization hit maximum capacity. Dependency Agent Failure propagation path identified: Gateway → Order Service → Payment Service → Redis Alert Agent Seven alerts detected. Five classified as downstream symptoms. Historical Agent 94% similarity to a previous Redis connection pool exhaustion incident. After correlating all evidence, the reasoning engine concluded: Root Cause Redis connection pool exhaustion. Confidence 96% Recommended Actions Total investigation time: Approximately 30 seconds. TattvaAI combines modern AI orchestration with production-ready backend engineering. To validate the platform end-to-end, I built an entire observable microservices environment. It includes: This allows TattvaAI to investigate realistic production incidents instead of synthetic examples. After running repeated investigations inside the demo environment, the results were encouraging. One principle guided the design of TattvaAI from the beginning. Every conclusion should be explainable. Instead of producing a single answer, every investigation includes: If TattvaAI concludes that Redis caused the outage, it also shows: The goal is transparency. Engineers should understand why the AI reached its conclusion. Although developed during a hackathon, TattvaAI was designed with production deployment in mind. It includes: The architecture is intentionally modular so organizations can extend it with additional agents and custom workflows. The observability ecosystem has matured significantly over the past decade. We no longer struggle to collect telemetry. We struggle to interpret it quickly enough when production is failing. TattvaAI shifts observability from passive monitoring to active investigation. Instead of showing engineers hundreds of charts, it answers the question they actually care about: What is broken, why did it break, and what should I do next? That shift has implications beyond reducing Mean Time to Resolution MTTR . It can also help teams: This project represents the first version of a much larger vision. Future directions include: Building TattvaAI fundamentally changed the way I think about observability. The future of incident response is not simply collecting more telemetry or building more dashboards. It is creating intelligent systems capable of reasoning across telemetry the same way experienced engineers do. SigNoz provides an excellent foundation for collecting and exposing observability data. TattvaAI builds on that foundation by transforming telemetry into explanations, hypotheses, and actionable decisions. If AI can reduce a two-hour investigation to thirty seconds while remaining transparent and explainable, then incident response becomes less about searching for information and more about solving problems. And that is exactly the future I wanted to build. TattvaAI was built for the SigNoz "Agents of SigNoz" Hackathon July 2026 . The project is open source, and I would love to hear feedback from the observability community. If you're interested in AI-powered incident investigation, explore the project, experiment with it, and let me know what improvements you'd like to see. Happy building—and may your next production incident take seconds instead of hours to understand.