The Architecture of Resilience #
The core philosophy here is that the safety net cannot be a point of failure. If your resilience system depends on an external API to perform a rollback, you've just added another way for the system to break.
I structured the deployment using a strict split:
Local Critical Path (SQLite): I used Node'snode:sqlite
for a Checkpoint Manager. Every agent step—including the plan, budget consumed, and completed actions—is written as a durable checkpoint to disk. Rollbacks pull from here exclusively.Decision Support (SigNoz): I integrated SigNoz for context and explanation. The Policy Engine queries the SigNoz Query API for prior-run context to score severity, and the Explanation Layer uses the SigNozMCPserver to ground natural-language explanations in actual trace data. If SigNoz is unreachable, the system degrades gracefully rather than crashing.
Turning Observability into Action #
Most people treat OpenTelemetry as a post-mortem tool, but for a real-world AI workflow, it needs to be load-bearing. I implemented this in three ways:
1. Correlated Tracing
Instead of simple logs, every meaningful event is a child span under a parent run.execute
span. This ensures a complete audit trail of the agent's logic.
const SpanNames = {
RUN_EXECUTE: "run.execute",
EXECUTOR_STEP: "executor.step",
CHECKPOINT_CREATED: "checkpoint.created",
POLICY_EVALUATION: "policy.evaluation",
POLICY_EXPLANATION: "policy.explanation",
CHECKPOINT_RESTORED: "checkpoint.restored",
} as const;
2. Mid-run Context Queries
The Policy Engine doesn't judge steps in isolation. It queries SigNoz during the run to see if previous scores have already crossed a threshold, escalating the severity if the agent is repeatedly failing.
if (prior.queried && prior.previousScores.some((s) => s >= config.Threshold)) {
// escalate — this isn't the agent's first warning this run
}
3. Dynamic Explanations via MCP
When a policy violation occurs, the system performs a JSON-RPC tools/call
to signoz_search_traces
. This replaces canned error messages with actual data from the trace.
[signoz-mcp] INVOKING tools/call signoz_search_traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp
By shifting from "monitoring after the fact" to "intervening during execution," we can actually deploy LLM agents in production without worrying about them quietly spending the entire budget on the wrong task.
Next Agents-Concerto: Orchestrating Claude Code for PRs →