Our incident-response agent got the root cause wrong 7 times out of 12. It still never made a bad rollback.

An engineer from the Agent K project reports that their incident-response agent correctly diagnosed root causes in only 5 out of 12 seeded production incidents, yet it never executed a bad rollback. The safety mechanism relies on deterministic code checks rather than model correctness, with a policy gate that denies actions if any of six checks fail.

We ran Agent K against 12 seeded production incidents. It named the correct root cause 5 times out of 12 — 3 out of 9 if you discard the runs we couldn't cleanly attribute. Over the same 12 runs it produced the correct rollback / no-rollback verdict 10 times , published zero claims without a SigNoz link behind them, and executed zero actions outside its policy gate. That gap between "diagnosed correctly" and "behaved correctly" is the entire project. Agent K was our entry for Agents of SigNoz WeMakeDevs × SigNoz , Track 01. The setup: a FastAPI RAG support service — 72 synthetic help-centre docs in Postgres + pgvector, local sentence-transformers embeddings, free-tier models behind one OpenAI-compatible client Groq by default; Cerebras, Gemini and OpenRouter swap in via one env var — instrumented with OpenTelemetry, shipping traces, metrics and logs to SigNoz over OTLP. We installed SigNoz with Foundry , so casting.yaml / casting.yaml.lock are committed and the stack is reproducible; pointing the same exporter at SigNoz Cloud later took no code change. When an alert fires, Agent K investigates through the SigNoz MCP server , publishes only claims carrying a resolvable SigNoz link, and rolls the service back only if a piece of plain Python says it may. We started from an assumption we never tried to engineer away: the model will be wrong. So correctness can't be the safety mechanism. The safety has to be code. Everything hangs off an in-process flag store with four seeded failures, and one deliberate asymmetry: app/flags.py FLAG NAMES: tuple str, ... = "prompt regression", "retry storm", "retrieval latency", "db pool exhaustion", The scenarios that represent a "deployment" a code/prompt change and therefore get a deployment.marker span; the other two are runtime/infra faults with no marker. DEPLOYMENT CLASS FLAGS: set str = {"prompt regression", "retry storm"} SigNoz has no native deployment-marker API we chased SigNoz/signoz 6162 https://github.com/SigNoz/signoz/issues/6162 , closed unshipped , so we emit our own deployment.marker OTel span through the pipeline that already exists. Two scenarios emit one; two don't. That present/absent asymmetry is the queryable signal — the absence of a marker is positive evidence that a rollback would not help. One POST /ask in SigNoz: the rag.retrieval, rag.prompt construction and chat spans, with the free SQLAlchemy DB span nested under retrieval. Six deterministic checks, all of which must pass: app/policy.py — THIS MODULE MUST NEVER CALL AN LLM. ACTION ALLOWLIST: tuple str, ... = "rollback", exactly one entry SANDBOX SERVICES = frozenset {"agent-k-rag-service"} BURN RATE THRESHOLD = 1.0 CONFIDENCE THRESHOLD = 0.70 COOLDOWN SECONDS = 600.0 slo breach , allowlist , cooldown , confidence , deployment related , sandbox scope . Every unknown fails closed — an alert we can't read a burn rate from parses to 0.0 , which fails the SLO check and denies the action. The "no LLM" part isn't a comment we trust ourselves to honour. One test walks the module's import graph with ast and fails the build if app.llm ever appears in it; another monkeypatches llm.generate to raise on any call and then runs a full evaluation, which catches an indirect call a grep would miss. Both live in the 219-test suite. This is the part worth reading twice. The deployment related check needs to know which scenario it's looking at, and the cheapest source was the model's claim text — our system prompt forces it to start its answer with one of the four scenario names. Fine, until the first 12-run eval printed this: two db pool exhaustion incidents narrated as retry storm , each approving a rollback that could not possibly have fixed a database pool problem. The gate wasn't broken. It was doing exactly what we wrote — and what we'd written was "the model may authorise a rollback by writing the word retry storm in a sentence." Every safety property we were claiming had a hole in it shaped like a string match. The fix was to make the check require corroboration from telemetry, not narration: app/policy.py — check 5 observed = frozenset deployment markers or is deployment class = incident type in DEPLOYMENT CLASS FLAGS marker seen = incident type in observed deployment ok = is deployment class and marker seen And deployment markers is only ever populated from a tool result, never from a claim: app/investigation.py — inside the evidence-gathering step if ev type == "deployment": inv.deployment markers seen.update name for name in FLAG NAMES if f'"{name}"' in content text We also made the ordering load-bearing — an investigation that stopped before the marker query could never approve anything — and asserted it at import rather than leaving it in a comment: assert DEPLOYMENT QUERY INDEX < MIN EVIDENCE ITERATIONS, "the deployment-marker query must fall within MIN EVIDENCE ITERATIONS, or the " "policy gate can never corroborate a deployment-class claim" After that fix, both bad approvals disappeared. The agent still misdiagnoses — it called retrieval latency "retry storm" in all three of those runs — but a misdiagnosis now produces a denial , not a rollback. The incident report's policy decision: five checks pass, deployment related fails, and the action is denied with a next step for the human. 1. SigNoz's OTLP receivers don't bind until you finish first-run setup. A freshly cast stack looks healthy — every container up, curl localhost:8080 returns 200 — while curl localhost:4318/v1/traces gives connection reset by peer , which reads exactly like a broken exporter. The collector is OpAMP-managed and can't register an agent without an org, so the backend loops on cannot create agent without orgId . Check it directly: curl -s http://localhost:8080/api/v1/version "setupCompleted":false Register an org UI or POST /api/v1/register and the receivers come up. We lost real time believing app/telemetry.py was wrong. It wasn't. 2. We invented MCP tool names. We'd written query traces , query logs , query metrics against a mental model of the API. The real inventory in signoz-mcp-server v0.9.0 is signoz search traces , signoz search logs , signoz aggregate traces . Every evidence query failed, stripping every claim and escalating every investigation — a total failure that looked like an agent problem. Two more traps followed: no Windows binary ships in the releases we cross-compiled in WSL , and SigNoz v0.134 rejects JWTs sent via SIGNOZ-API-KEY , wanting Authorization: Bearer — with a token that expires every 30 minutes, which is a delightful thing to discover mid-demo. 3. The free tier is a real design constraint. A 20-row trace search measured 8,424 tokens and came back HTTP 413 before a hypothesis existed. SigNoz returns every possible span attribute per row and most are null for this app, so we strip nulls rather than truncate — same signal, an order of magnitude fewer tokens. Then the model began diagnosing from the query engine's own statistics : "db pool exhaustion is likely due to an extremely high number of rows scanned 1018 and bytes scanned 10434 " . Those are the stats for Agent K's own query. Hence: ENGINE NOISE KEYS = frozenset { "meta", "rowsScanned", "bytesScanned", "durationMs", "stepIntervals", "nextCursor", "queryName", "columnType", "aggregationIndex", "signal", "fieldContext", "fieldDataType", "id", "warnings", } Evidence must describe the incident, never the act of looking at it. 4. The agent investigated our test suite. Our tests import app.main , which calls setup telemetry , which shipped every span pytest produced into the real SigNoz. A live run read deployment-marker counts of retry storm=7 vs prompt regression=3 when only prompt regression had been injected — the retry storm markers were pytest's — and misdiagnosed on that basis. AGENT K DISABLE OTLP EXPORT now suppresses export while still building all three providers. 5. A 200 from a single-page app is not a resolving link. We built evidence links to /deployments , a route that does not exist in SigNoz. Our link checker passed anyway — SigNoz is an SPA and answers 200 for every path, so a status check can't tell a real route from a typo. A human clicking the link landed on nothing. Deployment markers are spans, so the honest destination was the traces explorer. Our default base URL was wrong too: 3301 , SigNoz's older port, where Foundry serves on 8080 . Agent K never holds the Docker socket. A separate deployer sidecar does, and it exposes exactly one mutating endpoint that takes an empty body — the caller names no image, no service, no command. A caller that cannot name an image cannot be tricked into deploying an attacker's image. deployer: volumes: The privilege boundary, in one line. Nothing else in this project gets it. - /var/run/docker.sock:/var/run/docker.sock One rule in there took a failed rollback to learn: the command is docker compose up -d --force-recreate , never restart . restart brings the container back on its current image and reports success while changing nothing — the rollback appears to work and the incident continues. The Agent K dashboard in SigNoz: policy verdicts, why actions were denied, and recovery-verified counts side by side. The honest summary: the model is the least trustworthy component in the system, and designing around that made everything else simpler. All of this worked for our setup — one machine, one service, four failure modes we wrote ourselves. We haven't tested any of it against an incident we didn't author. Code: github.com/SujalXplores/Agent-K https://github.com/SujalXplores/Agent-K · Built for Agents of SigNoz https://www.wemakedevs.org/hackathons/signoz , Track 01. Built with AI coding assistance Claude Code , disclosed in the repo; the shipped agent itself runs on free-tier providers and local embeddings only.