Nights Watch: Guarding AI Agents Beyond the Wall

"Night gathers, and now my watch begins." The Night's Watch didn't exist to fight wars nobody saw coming — they existed because someone had to actually stand on the Wall and notice when something crossed it. That's the exact problem I kept running into with AI agents, and it's why I built Nights Watch for the "Agents of SigNoz" hackathon: a runtime resilience layer that catches an agent quietly drifting off its plan, explains why, and recovers — automatically.

Most agent failures aren't dramatic. An agent doesn't crash, it doesn't throw an exception, it doesn't get flagged by a content filter. It just... does something slightly different from what it was asked. Told to "find and book a flight under $400," a subtly-drifted agent might reason its way into a $1,200 upgrade and report back "done" — technically true, catastrophically wrong. Nothing in a normal observability stack notices this, because nothing failed. The agent succeeded at the wrong thing.

I wanted a system where SigNoz wasn't just a dashboard you check after something breaks — where it actively fed a decision-making loop while the agent was still running.

Everything else in the project falls out of one non-negotiable decision I made on day one: rollback state has to be local and durable, never dependent on an external service being reachable. If your resilience system's own safety net depends on a third-party API being up, you haven't built resilience, you've built a second point of failure.

So the split looks like this:

node:sqlite

. Rollback reads from here, always, no exceptions.That last part mattered more than I expected — more on that below.

It would've been easy to bolt OpenTelemetry onto an agent and call it "observability." I wanted SigNoz doing real work in three specific places.

1. Every run is one correlated trace. run.execute

is the parent span, and every meaningful event — executor.step

, checkpoint.created

, policy.evaluation

, checkpoint.restored

, policy.explanation

— is a real child span sharing that trace context, not just a log line:

const SpanNames = {
  RUN_EXECUTE: "run.execute",
  EXECUTOR_STEP: "executor.step",
  CHECKPOINT_CREATED: "checkpoint.created",
  POLICY_EVALUATION: "policy.evaluation",
  POLICY_EXPLANATION: "policy.explanation",
  CHECKPOINT_RESTORED: "checkpoint.restored",
} as const;

2. The Policy Engine actually queries SigNoz mid-run. Before scoring a step, it checks whether prior scores in the current trace already crossed the threshold — so severity isn't judged in isolation, it's judged against the run's actual history:

if (prior.queried && prior.previousScores.some((s) => s >= config.Threshold)) {
  // escalate — this isn't the agent's first warning this run
}

3. The Explanation Layer calls SigNoz's MCP server for real, not a canned template. When a violation fires, it does an actual JSON-RPC tools/call

to signoz_search_traces

, and I made sure to log the attempt distinctly so I could prove — to myself, and now to you — that it wasn't quietly falling back:

[signoz-mcp] INVOKING tools/call signoz_search_traces run=run-872ac2d2 endpoint=http://localhost:8000/mcp

The moment that made the whole project click for me was running the drift scenario live and watching the loop close. Here's real output from one of those runs — no editing:

[happy-path] runId=run-bdbc8d0b maxBudget=$400
[happy-path] search → 3 options flt-nw-101=$189, flt-nw-202=$249, flt-nw-303=$319
[happy-path] status=awaiting_approval score=40
[happy-path] recovery: recovered via rollback_replan; restored=run-bdbc8d0b-cp-1
[happy-path] explanation mcpInvoked=true mcpOk=true: The agent planned to keep
  spend under $400, but step "select" used tool select_flight at $1200
  (policy score 40). Rules fired: costUsd=1200 exceeds maxBudget=400.
[happy-path] done — status=completed recoveries=1 recovered=true

Score spiked to 40 the instant the $1,200 option got selected, the run d for a human decision (Phase 7 added an actual approve/reject gate — I didn't want a system that silently overrides a human either), and rejecting it rolled the agent back to the last safe checkpoint, re-planned with the violation added as explicit context, and finished the job correctly at $189.

Building the Recovery Engine taught me not to trust my own logs. It's easy to print "recovered successfully" and believe it. So I built a manual verification ritual: pull the local checkpoint list, then open the same run in SigNoz's Traces UI, and check that every checkpoint.created

and checkpoint.restored

span's attributes actually match the SQLite rows — same checkpoint.id

, same budgetConsumed

, same planStep

.

checkpoint.id            budgetConsumed   label
run-...-cp-2              1200            option_selected   ← the violation
run-...-cp-3               189            option_selected   ← post-recovery

Watching those two independent sources of truth — my own database and SigNoz's traces — agree, span by span, was more convincing than any unit test. That's also where I found and fixed a real bug: an n8n workflow node that picked options[0]

instead of the cheapest flight, which would've silently selected the injected $1,200 "drift bait" flight if drift injection order ever changed. SigNoz didn't catch that bug — the habit of cross-checking against it did.

I self-hosted SigNoz under WSL2 on Windows, and lost real hours to something that had nothing to do with agents or observability logic: Windows' port-forwarding to WSL goes stale every time WSL restarts, and WSL restarts on its own after idle periods. localhost:8080

would just... stop connecting, with no useful error beyond a generic connection reset. The fix was mundane (re-run a script that refreshes the Windows-to-WSL port mapping after every restart) but the lesson wasn't: the fanciest part of your system is rarely what breaks first — the boring plumbing under it is. I used SigNoz's Foundry installer (foundryctl cast -f casting.yaml

) to at least make the SigNoz side of that setup reproducible, which I'd recommend to anyone self-hosting for a demo.

The other thing I'd do differently: I'd build the pacing into the demo executor earlier. My first automated demo endpoint fired all four agent steps back-to-back in milliseconds — technically correct, but useless for actually watching the drift-and-recovery moment happen on a dashboard. A deliberate ~1 second delay between steps turned out to matter more for demoing resilience than any amount of extra Policy Engine logic.

The full loop — plan, execute, drift, detect, explain, for approval, roll back, re-plan, recover — runs end to end, with SigNoz doing real decision-support work at two separate points, not just recording what happened after the fact. The deployed app link runs the complete agent loop with graceful degradation in place of live SigNoz calls (SigNoz itself stays self-hosted locally, by the same "don't put your safety net on an external dependency" logic that shaped the whole architecture); the trace-level verification lives in the project's demo video and this post.

If there's one thing I'd want another builder to take from this: instrument for the failure you're actually worried about, not the one that's easy to instrument. Uptime and error rates are easy. "Is my agent quietly doing the wrong thing while reporting success" is hard — and it's the one that actually needed SigNoz sitting in the decision loop, not just watching from the sidelines.

Night gathers. Someone should still be watching.