{"slug": "evidence-first-answer-second-building-an-observable-industrial-ai-agent-with", "title": "Evidence First, Answer Second: Building an Observable Industrial AI Agent with SigNoz", "summary": "A developer building an Industrial IoT Anomaly Control system for a simulated water-treatment plant created an observable AI agent that stops, shows evidence, and escalates to humans when confidence is low. The system uses SigNoz to make the decision process visible, separating safety logic from the LLM to prevent dangerous AI actions on the factory floor.", "body_md": "Most AI systems are designed to give an answer.\n\nThat is useful in a chatbot. On a factory floor, it can be dangerous.\n\nWhile building **Industrial IoT Anomaly Control**, I kept coming back to one question:\n\nWhat should an AI agent do when it detects a real problem but does not have enough evidence to explain the cause safely?\n\nMy answer was simple: it should stop, show what it knows, and send the case to a human.\n\nThis project is a real-time monitoring system for a simulated water-treatment plant. It streams live sensor data from six industrial assets, detects unusual behaviour, searches a knowledge base for similar incidents, and then chooses between two paths:\n\nSigNoz is what makes this decision process visible. Instead of seeing only the final AI response, I can inspect the full path from the incoming sensor reading to anomaly detection, knowledge retrieval, policy checks, agent explanation, and recovery.\n\nFactories already collect large amounts of telemetry such as vibration, temperature, humidity, sequence numbers, and timestamps.\n\nThe problem is not missing data. The problem is turning that data into a useful decision.\n\nA traditional threshold alarm may say:\n\nVibration is above the configured limit.\n\nThat still leaves the operator with several questions:\n\nRepeated threshold alerts can also create alarm fatigue. If one fault produces dozens of alerts, operators may start treating them as noise.\n\nI wanted the system to create one investigation instead of another flood of alarms.\n\nThe demo represents six assets in a water-treatment plant. A TCP simulator can run in healthy mode or a seeded faulty mode that creates repeatable equipment and data-quality problems.\n\nThe system detects:\n\nOne design decision became very important: **bad telemetry is not the same as a broken machine**. Duplicate events and sequence gaps become data-quality incidents and can never turn into mechanical recommendations.\n\nRelated abnormal readings are grouped into one investigation, avoiding a new alert for every reading.\n\nWhen the system detects an equipment-condition incident, it searches a Chroma knowledge base containing structured water-treatment scenarios.\n\nThe search is filtered before similarity matching. Equipment type, sensor type, and incident category are used to prevent unrelated incidents from becoming evidence.\n\nFor example, a vibration problem on a centrifugal pump should not be explained using a humidity issue from a different asset.\n\nThe retrieval step returns the strongest matches, similarity scores, verification status, and the gap between the first and second result. This is where my project differs from a basic RAG application.\n\nA traditional RAG flow often looks like this:\n\n```\nQuestion → Retrieve documents → Generate an answer\n```\n\nMy flow is closer to:\n\n```\nAnomaly → Collect evidence → Retrieve incidents → Check confidence\n        → Recommend or escalate → Explain the approved result\n```\n\nThe agent is not allowed to treat every retrieved document as enough evidence.\n\nI did not ask the model to generate a confidence number.\n\nA language model can return “95% confident,” but that number may not represent a tested probability. For an industrial use case, that is not a safe decision rule.\n\nInstead, the application calculates confidence from measurable signals:\n\nThe main safety rule is:\n\n```\nassert not (\n    decision == \"RECOMMEND\"\n    and confidence < configured_threshold\n)\n```\n\nIf the evidence passes policy, the system recommends a bounded inspection action. If the match is weak, unclear, unverified, or missing, the decision becomes `ESCALATE`\n\n.\n\nThe LLM only explains the result. It cannot override policy or invent evidence. If the provider is unavailable, a deterministic fallback answers while sensor processing continues.\n\nThat separation was one of my biggest lessons from the project:\n\nThe application should own safety. The model should help communicate it.\n\nAdding observability meant moving beyond “the API is up” and “the model returned 200.” I wanted to observe the quality of the agent’s decision.\n\nThe application emits OpenTelemetry traces, metrics, and structured incident logs through OTLP HTTP, keeping monitoring separate from business logic.\n\nA normal reading creates a root span named:\n\n```\nsensor.process\n```\n\nAn anomalous investigation can include:\n\n```\nsensor.process\n├── detectors.evaluate\n├── incident.evaluate\n│   └── policy.evaluate\n├── knowledge.enrich_incident\n│   ├── knowledge.retrieve\n│   └── policy.re_evaluate\n├── agent.explain\n└── agent.monitor_recovery\n```\n\nThe first policy decision is created immediately. Retrieval and explanation run in the background, because a slow embedding or LLM call must never pause the live stream.\n\nThe trace connects details such as:\n\nRaw measurements, prompts, keys, and repair notes are deliberately excluded from telemetry.\n\nI created a SigNoz dashboard called **Water Treatment Agent — Trust & Recovery**.\n\nThe most useful panels are:\n\nOne real issue I hit was using the wrong metric reduction for automatic resolutions. A rate on a total-count panel produced an empty or confusing result. Using an increase across the selected range and displaying the sum fixed it.\n\nI learned this by watching real telemetry, not by only reading documentation.\n\nI also grouped agent latency by both the attempted mode and final mode.\n\nThat lets me distinguish:\n\nA fallback is therefore a visible operational signal, not a hidden log line.\n\nThe operator dashboard stores the W3C trace ID with each incident.\n\nFrom the active incident, the operator can click **Open trace in SigNoz** and jump directly to the full investigation.\n\nThis connects the product directly to observability:\n\n```\nOperator sees incident\n→ Opens exact trace\n→ Checks detector evidence\n→ Checks retrieved knowledge\n→ Checks policy decision\n→ Checks model or fallback latency\n→ Watches recovery\n```\n\nThe agent does not stop after generating an explanation.\n\nIt keeps watching the asset. An equipment incident is automatically resolved only after five consecutive healthy readings. The recovery span records the current healthy count and whether the incident was safely closed.\n\nOperators can record the real finding and fix. It becomes clearly labelled local knowledge without replacing verified evidence.\n\nThe biggest lesson was that agent observability is not only token counts and model latency. Those metrics do not answer the most important question:\n\nWhy did the system believe it had enough evidence to recommend an action?\n\nMeaningful observability required connecting:\n\nAbstention should also be visible. Escalation is not a failed response; it is often the safer outcome. SigNoz shows how often the agent abstains and whether retrieval or data quality caused it.\n\nI also kept the live path independent from slow AI work. The safe detector-only result reaches the operator first, then retrieval and explanation enrich it.\n\nIndustrial IoT Anomaly Control is not designed to always sound confident.\n\nIt is designed to build an evidence chain, follow clear rules, and stop when that evidence is not strong enough.\n\nSigNoz turns that evidence chain into something an operator or engineer can inspect—from the first unusual reading to the final decision and recovery.\n\nThe principle behind the project is simple:\n\nEvidence first. Answer second.", "url": "https://wpnews.pro/news/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with", "canonical_source": "https://dev.to/inraj/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with-signoz-4jc5", "published_at": "2026-07-26 00:51:21+00:00", "updated_at": "2026-07-26 01:00:16.058310+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "ai-safety"], "entities": ["SigNoz", "Chroma", "Industrial IoT Anomaly Control"], "alternates": {"html": "https://wpnews.pro/news/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with", "markdown": "https://wpnews.pro/news/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with.md", "text": "https://wpnews.pro/news/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with.txt", "jsonld": "https://wpnews.pro/news/evidence-first-answer-second-building-an-observable-industrial-ai-agent-with.jsonld"}}