# Nights Watch: A Runtime Resilience Layer for AI Agents

> Source: <https://promptcube3.com/en/threads/3738/>
> Published: 2026-07-26 15:47:20+00:00

# Nights Watch: A Runtime Resilience Layer for AI Agents

## The Architecture of Resilience

The core philosophy here is that the safety net cannot be a point of failure. If your resilience system depends on an external API to perform a rollback, you've just added another way for the system to break.

I structured the deployment using a strict split:

**Local Critical Path (SQLite):** I used Node's`node:sqlite`

for a Checkpoint Manager. Every agent step—including the plan, budget consumed, and completed actions—is written as a durable checkpoint to disk. Rollbacks pull from here exclusively.**Decision Support (SigNoz):** I integrated SigNoz for context and explanation. The Policy Engine queries the SigNoz Query API for prior-run context to score severity, and the Explanation Layer uses the SigNoz[MCP](/en/tags/mcp/)server to ground natural-language explanations in actual trace data. If SigNoz is unreachable, the system degrades gracefully rather than crashing.

## Turning Observability into Action

Most people treat OpenTelemetry as a post-mortem tool, but for a real-world AI workflow, it needs to be load-bearing. I implemented this in three ways:

**1. Correlated Tracing**

Instead of simple logs, every meaningful event is a child span under a parent `run.execute`

span. This ensures a complete audit trail of the agent's logic.

``` js
const SpanNames = {
 RUN_EXECUTE: "run.execute",
 EXECUTOR_STEP: "executor.step",
 CHECKPOINT_CREATED: "checkpoint.created",
 POLICY_EVALUATION: "policy.evaluation",
 POLICY_EXPLANATION: "policy.explanation",
 CHECKPOINT_RESTORED: "checkpoint.restored",
} as const;
```

**2. Mid-run Context Queries**

The Policy Engine doesn't judge steps in isolation. It queries SigNoz during the run to see if previous scores have already crossed a pause threshold, escalating the severity if the agent is repeatedly failing.

``` js
if (prior.queried && prior.previousScores.some((s) => s >= config.pauseThreshold)) {
 // escalate — this isn't the agent's first warning this run
}
```

**3. Dynamic Explanations via MCP**

When a policy violation occurs, the system performs a JSON-RPC `tools/call`

to `signoz_search_traces`

. This replaces canned error messages with actual data from the trace.

```
[signoz-mcp] INVOKING tools/call signoz_search_traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp
```

By shifting from "monitoring after the fact" to "intervening during execution," we can actually deploy LLM agents in production without worrying about them quietly spending the entire budget on the wrong task.

[Next Agents-Concerto: Orchestrating Claude Code for PRs →](/en/threads/3729/)