{"slug": "building-missio-an-evidence-bound-remediation-agent-with-signoz", "title": "Building Missio: An Evidence-Bound Remediation Agent with SigNoz", "summary": "A developer built Missio, an evidence-bound remediation agent for the Agents of SigNoz hackathon. Missio starts with a real incident reported in Slack, reconstructs what happened from SigNoz, finds the exact source revision that produced the failure, asks an agent for a tightly bounded change, and opens a draft pull request only after a human approves the action. The system stops if evidence is missing, if GitHub cannot resolve the production revision, or if the model returns an unsafe patch.", "body_md": "A production error is exactly when an AI agent should be least creative.\n\nThat sentence became the design rule behind Missio, a terminal mission control system I built for the Agents of SigNoz hackathon. Missio starts with a real incident reported in Slack, reconstructs what happened from SigNoz, finds the exact source revision that produced the failure, asks an agent for a tightly bounded change, and opens a draft pull request only after a human approves the action.\n\nThe entire journey stays visible in one Rust TUI:\n\n```\nSIGNAL → EVIDENCE → CODE → PATCH → HUMAN → PR → CI → HANDOFF\n```\n\nThis is not a dashboard with an AI summary beside it. Every stage represents something the system has actually observed or completed. If evidence is missing, the mission stops. If GitHub cannot\n\nresolve the production revision, it stops. If the model returns an unsafe patch, it stops.\n\nObservability tools are already good at showing that a request failed. The harder question is what an agent should be allowed to do next.\n\nAn error message alone does not establish which code was deployed. A repository's default branch may have moved since the failure. A model can produce a convincing explanation using the wrong version of a file. It can also quietly fill missing context with assumptions.\n\nI wanted the opposite behavior. Missio should be useful only when it can build a chain from a production signal to a specific piece of source code. The agent's job is to reason inside that chain, not invent one.\n\nThat made SigNoz the evidence system of record rather than a decorative integration.\n\nFor the demo, I use an instrumented checkout service. A bad discount request reaches a real HTTP endpoint and produces a failing OpenTelemetry span. The span is exported to SigNoz with the normal service and error fields, plus the details Missio needs to connect runtime behavior to source:\n\nThen an operator opens the incident from an allowlisted Slack channel:\n\n```\n/missio-sev SEV1 service production/checkout checkout requests are failing\n```\n\nSlack is only the admission point. The message does not tell the agent what the root cause is, and Missio does not treat the wording as evidence. It uses the resource and incident window to query the corresponding telemetry in SigNoz.\n\nThe most useful lesson from building Missio did not come from reading documentation. It came from a demo that failed even though every individual component appeared healthy.\n\nThe checkout service successfully exported its span. SigNoz accepted the telemetry. But Missio queried immediately and received no matching rows. A short time later, the same query returned the failure exactly as expected.\n\nIngestion acceptance and query availability are not the same moment.\n\nThat distinction sounds small, but it changes the behavior of an autonomous workflow. If an empty first response means “there was no incident,” the agent can discard a genuine production failure.\n\nIf the system waits forever, the operator cannot tell whether it is making progress.\n\nMissio now performs bounded evidence acquisition and makes the wait visible in the TUI. It retries for a limited period while SigNoz indexes the span. If a valid evidence bundle still cannot be formed, the EVIDENCE stage fails closed. It never replaces missing telemetry with a fabricated\n\nincident just to keep the animation moving.\n\nThis was also why I kept the signal horizon at the bottom of the interface. The operator can see whether Missio is listening, acquiring evidence, or holding because the observation is incomplete.\n\nOnce the trace is available, Missio extracts the repository and the full deployed commit from the span. That commit matters more than the current state of the default branch: it identifies the code that actually produced the observed failure.\n\nMissio reads the repository tree at that immutable revision and selects source linked to the observed code location. In a monorepo, it can keep the investigation inside the affected service. Only that bounded context reaches the agent.\n\nThis gives the model a much more honest question:\n\nGiven this observed failure and the exact source that produced it, is there enough evidence for a small fix?\n\nThe model can either hold the mission or propose replacement content for source it was given. It cannot add an unrelated file, edit the CI workflow, reach outside the service boundary, or claim that an external action succeeded. Those decisions are enforced by Rust after the model responds.\n\nMissio uses Rig for the agent runtime and an explicitly configured model through OpenRouter. The model never receives Slack, SigNoz, or GitHub credentials.\n\nI originally thought of the interface as the visual identity of the project: a mission-control orbit instead of another grid of cards and tables. While building the real workflow, it became something more useful.\n\nThe TUI shows who currently has authority.\n\nSigNoz can provide evidence. The agent can propose a patch. Neither can create a pull request. When a proposed change passes policy, the mission pauses at HUMAN and shows the target repository. The operator must explicitly approve one draft-PR attempt.\n\nThat approval is deliberately narrow. It does not authorize a merge or deployment. If GitHub rejects the request, Missio preserves the mission and returns control to the human instead of silently repeating a mutation.\n\nAfter approval, Missio creates a branch and draft pull request from the production revision identified by SigNoz. The evidence digest travels into the pull request, connecting the proposed change back to the incident that caused it.\n\nMissio then watches GitHub CI for the exact draft commit. When the checks pass, the final state is HANDOFF.\n\nThat wording is intentional. Missio does not pretend that a green check means production has recovered. It tells the operator that an evidence-bound draft is ready for normal review, merge, and deployment. The pull request remains the place where engineers can inspect the change and make\n\nthe final call.\n\nI considered extending the demo through automatic deployment, but it would have weakened the product's trust model. A hackathon animation should not manufacture authority that the real system\n\ndoes not have.\n\nWithout SigNoz, this would be a code agent triggered by a Slack message. With SigNoz, the agent is grounded in an observed resource, time window, trace, code location, repository, and deployed revision.\n\nThat context affects every later step:\n\nThe deepest lesson for me was that agent observability is not only about tracing what a model did. Observability can also control what the model is allowed to do next.\n\nMissio still leaves the final operational decision with a person. Its job is to turn a noisy incident into a reviewable, evidence-bound change without hiding the gaps along the way.\n\nYou can explore [Missio on GitHub](https://github.com/N-45div/Missio).\n\nIf you want to trace the pieces I built on, the most useful references were the SigNoz guides for", "url": "https://wpnews.pro/news/building-missio-an-evidence-bound-remediation-agent-with-signoz", "canonical_source": "https://dev.to/n45div/building-missio-an-evidence-bound-remediation-agent-with-signoz-47", "published_at": "2026-07-26 11:51:55+00:00", "updated_at": "2026-07-26 12:30:20.800598+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-safety"], "entities": ["Missio", "SigNoz", "Slack", "GitHub", "Rust"], "alternates": {"html": "https://wpnews.pro/news/building-missio-an-evidence-bound-remediation-agent-with-signoz", "markdown": "https://wpnews.pro/news/building-missio-an-evidence-bound-remediation-agent-with-signoz.md", "text": "https://wpnews.pro/news/building-missio-an-evidence-bound-remediation-agent-with-signoz.txt", "jsonld": "https://wpnews.pro/news/building-missio-an-evidence-bound-remediation-agent-with-signoz.jsonld"}}