{"slug": "nights-watch-a-runtime-resilience-layer-for-ai-agents", "title": "Nights Watch: A Runtime Resilience Layer for AI Agents", "summary": "A developer has built Nights Watch, a runtime resilience layer for AI agents that uses a local SQLite checkpoint manager for rollbacks and SigNoz for decision support, ensuring the system degrades gracefully if SigNoz is unreachable. The architecture treats OpenTelemetry as a load-bearing component by implementing correlated tracing, mid-run context queries, and dynamic explanations via MCP to enable real-time intervention during agent execution.", "body_md": "# Nights Watch: A Runtime Resilience Layer for AI Agents\n\n## The Architecture of Resilience\n\nThe core philosophy here is that the safety net cannot be a point of failure. If your resilience system depends on an external API to perform a rollback, you've just added another way for the system to break.\n\nI structured the deployment using a strict split:\n\n**Local Critical Path (SQLite):** I used Node's`node:sqlite`\n\nfor a Checkpoint Manager. Every agent step—including the plan, budget consumed, and completed actions—is written as a durable checkpoint to disk. Rollbacks pull from here exclusively.**Decision Support (SigNoz):** I integrated SigNoz for context and explanation. The Policy Engine queries the SigNoz Query API for prior-run context to score severity, and the Explanation Layer uses the SigNoz[MCP](/en/tags/mcp/)server to ground natural-language explanations in actual trace data. If SigNoz is unreachable, the system degrades gracefully rather than crashing.\n\n## Turning Observability into Action\n\nMost people treat OpenTelemetry as a post-mortem tool, but for a real-world AI workflow, it needs to be load-bearing. I implemented this in three ways:\n\n**1. Correlated Tracing**\n\nInstead of simple logs, every meaningful event is a child span under a parent `run.execute`\n\nspan. This ensures a complete audit trail of the agent's logic.\n\n``` js\nconst SpanNames = {\n RUN_EXECUTE: \"run.execute\",\n EXECUTOR_STEP: \"executor.step\",\n CHECKPOINT_CREATED: \"checkpoint.created\",\n POLICY_EVALUATION: \"policy.evaluation\",\n POLICY_EXPLANATION: \"policy.explanation\",\n CHECKPOINT_RESTORED: \"checkpoint.restored\",\n} as const;\n```\n\n**2. Mid-run Context Queries**\n\nThe Policy Engine doesn't judge steps in isolation. It queries SigNoz during the run to see if previous scores have already crossed a pause threshold, escalating the severity if the agent is repeatedly failing.\n\n``` js\nif (prior.queried && prior.previousScores.some((s) => s >= config.pauseThreshold)) {\n // escalate — this isn't the agent's first warning this run\n}\n```\n\n**3. Dynamic Explanations via MCP**\n\nWhen a policy violation occurs, the system performs a JSON-RPC `tools/call`\n\nto `signoz_search_traces`\n\n. This replaces canned error messages with actual data from the trace.\n\n```\n[signoz-mcp] INVOKING tools/call signoz_search_traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp\n```\n\nBy shifting from \"monitoring after the fact\" to \"intervening during execution,\" we can actually deploy LLM agents in production without worrying about them quietly spending the entire budget on the wrong task.\n\n[Next Agents-Concerto: Orchestrating Claude Code for PRs →](/en/threads/3729/)", "url": "https://wpnews.pro/news/nights-watch-a-runtime-resilience-layer-for-ai-agents", "canonical_source": "https://promptcube3.com/en/threads/3738/", "published_at": "2026-07-26 15:47:20+00:00", "updated_at": "2026-07-26 16:16:14.742781+00:00", "lang": "en", "topics": ["ai-agents", "ai-safety", "ai-infrastructure", "developer-tools"], "entities": ["Nights Watch", "SQLite", "SigNoz", "OpenTelemetry", "MCP"], "alternates": {"html": "https://wpnews.pro/news/nights-watch-a-runtime-resilience-layer-for-ai-agents", "markdown": "https://wpnews.pro/news/nights-watch-a-runtime-resilience-layer-for-ai-agents.md", "text": "https://wpnews.pro/news/nights-watch-a-runtime-resilience-layer-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/nights-watch-a-runtime-resilience-layer-for-ai-agents.jsonld"}}