Nights Watch: A Runtime Resilience Layer for AI Agents

A developer has built Nights Watch, a runtime resilience layer for AI agents that uses a local SQLite checkpoint manager for rollbacks and SigNoz for decision support, ensuring the system degrades gracefully if SigNoz is unreachable. The architecture treats OpenTelemetry as a load-bearing component by implementing correlated tracing, mid-run context queries, and dynamic explanations via MCP to enable real-time intervention during agent execution.

Nights Watch: A Runtime Resilience Layer for AI Agents The Architecture of Resilience The core philosophy here is that the safety net cannot be a point of failure. If your resilience system depends on an external API to perform a rollback, you've just added another way for the system to break. I structured the deployment using a strict split: Local Critical Path SQLite : I used Node's node:sqlite for a Checkpoint Manager. Every agent step—including the plan, budget consumed, and completed actions—is written as a durable checkpoint to disk. Rollbacks pull from here exclusively. Decision Support SigNoz : I integrated SigNoz for context and explanation. The Policy Engine queries the SigNoz Query API for prior-run context to score severity, and the Explanation Layer uses the SigNoz MCP /en/tags/mcp/ server to ground natural-language explanations in actual trace data. If SigNoz is unreachable, the system degrades gracefully rather than crashing. Turning Observability into Action Most people treat OpenTelemetry as a post-mortem tool, but for a real-world AI workflow, it needs to be load-bearing. I implemented this in three ways: 1. Correlated Tracing Instead of simple logs, every meaningful event is a child span under a parent run.execute span. This ensures a complete audit trail of the agent's logic. js const SpanNames = { RUN EXECUTE: "run.execute", EXECUTOR STEP: "executor.step", CHECKPOINT CREATED: "checkpoint.created", POLICY EVALUATION: "policy.evaluation", POLICY EXPLANATION: "policy.explanation", CHECKPOINT RESTORED: "checkpoint.restored", } as const; 2. Mid-run Context Queries The Policy Engine doesn't judge steps in isolation. It queries SigNoz during the run to see if previous scores have already crossed a pause threshold, escalating the severity if the agent is repeatedly failing. js if prior.queried && prior.previousScores.some s = s = config.pauseThreshold { // escalate — this isn't the agent's first warning this run } 3. Dynamic Explanations via MCP When a policy violation occurs, the system performs a JSON-RPC tools/call to signoz search traces . This replaces canned error messages with actual data from the trace. signoz-mcp INVOKING tools/call signoz search traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp By shifting from "monitoring after the fact" to "intervening during execution," we can actually deploy LLM agents in production without worrying about them quietly spending the entire budget on the wrong task. Next Agents-Concerto: Orchestrating Claude Code for PRs → /en/threads/3729/