# Our incident-response agent got the root cause wrong 7 times out of 12. It still never made a bad rollback. > Source: > Published: 2026-07-25 20:54:35+00:00 We ran Agent K against 12 seeded production incidents. It named the correct root cause **5 times out of 12** — 3 out of 9 if you discard the runs we couldn't cleanly attribute. Over the same 12 runs it produced the **correct rollback / no-rollback verdict 10 times**, published zero claims without a SigNoz link behind them, and executed zero actions outside its policy gate. That gap between "diagnosed correctly" and "behaved correctly" is the entire project. Agent K was our entry for **Agents of SigNoz** (WeMakeDevs × SigNoz), Track 01. The setup: a FastAPI RAG support service — 72 synthetic help-centre docs in Postgres + pgvector, local `sentence-transformers` embeddings, free-tier models behind one OpenAI-compatible client (Groq by default; Cerebras, Gemini and OpenRouter swap in via one env var) — instrumented with OpenTelemetry, shipping traces, metrics and logs to SigNoz over OTLP. We installed SigNoz with **Foundry**, so `casting.yaml` /`casting.yaml.lock` are committed and the stack is reproducible; pointing the same exporter at SigNoz Cloud later took no code change. When an alert fires, Agent K investigates through the **SigNoz MCP server**, publishes only claims carrying a resolvable SigNoz link, and rolls the service back **only** if a piece of plain Python says it may. We started from an assumption we never tried to engineer away: the model will be wrong. So correctness can't be the safety mechanism. The safety has to be code. Everything hangs off an in-process flag store with four seeded failures, and one deliberate asymmetry: ``` # app/flags.py FLAG_NAMES: tuple[str, ...] = ( "prompt_regression", "retry_storm", "retrieval_latency", "db_pool_exhaustion", ) # The scenarios that represent a "deployment" (a code/prompt change) and therefore # get a deployment.marker span; the other two are runtime/infra faults with no marker. DEPLOYMENT_CLASS_FLAGS: set[str] = {"prompt_regression", "retry_storm"} ``` SigNoz has no native deployment-marker API (we chased [SigNoz/signoz#6162](https://github.com/SigNoz/signoz/issues/6162), closed unshipped), so we emit our own `deployment.marker` OTel span through the pipeline that already exists. Two scenarios emit one; two don't. **That present/absent asymmetry is the queryable signal** — the absence of a marker is positive evidence that a rollback would not help. *One POST /ask in SigNoz: the rag.retrieval, rag.prompt_construction and chat spans, with the free SQLAlchemy DB span nested under retrieval.* Six deterministic checks, all of which must pass: ``` # app/policy.py — THIS MODULE MUST NEVER CALL AN LLM. ACTION_ALLOWLIST: tuple[str, ...] = ("rollback",) # exactly one entry SANDBOX_SERVICES = frozenset({"agent-k-rag-service"}) BURN_RATE_THRESHOLD = 1.0 CONFIDENCE_THRESHOLD = 0.70 COOLDOWN_SECONDS = 600.0 ``` `slo_breach` , `allowlist` , `cooldown` , `confidence` , `deployment_related` , `sandbox_scope` . Every unknown fails closed — an alert we can't read a burn rate from parses to `0.0` , which fails the SLO check and denies the action. The "no LLM" part isn't a comment we trust ourselves to honour. One test walks the module's import graph with `ast` and fails the build if `app.llm` ever appears in it; another monkeypatches `llm.generate` to raise on any call and then runs a full evaluation, which catches an indirect call a grep would miss. Both live in the 219-test suite. This is the part worth reading twice. The `deployment_related` check needs to know which scenario it's looking at, and the cheapest source was the model's claim text — our system prompt forces it to start its answer with one of the four scenario names. Fine, until the first 12-run eval printed this: two `db_pool_exhaustion` incidents narrated as `retry_storm` , each **approving a rollback** that could not possibly have fixed a database pool problem. The gate wasn't broken. It was doing exactly what we wrote — and what we'd written was *"the model may authorise a rollback by writing the word retry_storm in a sentence."* Every safety property we were claiming had a hole in it shaped like a string match. The fix was to make the check require corroboration from telemetry, not narration: ``` # app/policy.py — check 5 observed = frozenset(deployment_markers or ()) is_deployment_class = incident_type in DEPLOYMENT_CLASS_FLAGS marker_seen = incident_type in observed deployment_ok = is_deployment_class and marker_seen ``` And `deployment_markers` is only ever populated from a tool result, never from a claim: ``` # app/investigation.py — inside the evidence-gathering step if ev_type == "deployment": inv.deployment_markers_seen.update( name for name in FLAG_NAMES if f'"{name}"' in content_text ) ``` We also made the ordering load-bearing — an investigation that stopped before the marker query could never approve anything — and asserted it at import rather than leaving it in a comment: ``` assert _DEPLOYMENT_QUERY_INDEX < MIN_EVIDENCE_ITERATIONS, ( "the deployment-marker query must fall within MIN_EVIDENCE_ITERATIONS, or the " "policy gate can never corroborate a deployment-class claim" ) ``` After that fix, both bad approvals disappeared. The agent still misdiagnoses — it called `retrieval_latency` "retry_storm" in all three of those runs — but a misdiagnosis now produces a *denial*, not a rollback. *The incident report's policy decision: five checks pass, deployment_related fails, and the action is denied with a next step for the human.* **1. SigNoz's OTLP receivers don't bind until you finish first-run setup.** A freshly cast stack looks healthy — every container up, `curl localhost:8080` returns 200 — while `curl localhost:4318/v1/traces` gives *connection reset by peer*, which reads exactly like a broken exporter. The collector is OpAMP-managed and can't register an agent without an org, so the backend loops on `cannot create agent without orgId` . Check it directly: ``` curl -s http://localhost:8080/api/v1/version # "setupCompleted":false ``` Register an org (UI or `POST /api/v1/register` ) and the receivers come up. We lost real time believing `app/telemetry.py` was wrong. It wasn't. **2. We invented MCP tool names.** We'd written `query_traces` , `query_logs` , `query_metrics` against a mental model of the API. The real inventory in signoz-mcp-server v0.9.0 is `signoz_search_traces` , `signoz_search_logs` , `signoz_aggregate_traces` . Every evidence query failed, stripping every claim and escalating every investigation — a total failure that looked like an agent problem. Two more traps followed: no Windows binary ships in the releases (we cross-compiled in WSL), and SigNoz v0.134 rejects JWTs sent via `SIGNOZ-API-KEY` , wanting `Authorization: Bearer` — with a token that expires every 30 minutes, which is a delightful thing to discover mid-demo. **3. The free tier is a real design constraint.** A 20-row trace search measured **8,424 tokens** and came back HTTP 413 before a hypothesis existed. SigNoz returns every possible span attribute per row and most are null for this app, so we strip nulls rather than truncate — same signal, an order of magnitude fewer tokens. Then the model began diagnosing from the *query engine's own statistics*: `"db_pool_exhaustion is likely due to an extremely high number of rows scanned (1018) and bytes scanned (10434)"` . Those are the stats for Agent K's own query. Hence: ``` _ENGINE_NOISE_KEYS = frozenset({ "meta", "rowsScanned", "bytesScanned", "durationMs", "stepIntervals", "nextCursor", "queryName", "columnType", "aggregationIndex", "signal", "fieldContext", "fieldDataType", "id", "warnings", }) ``` Evidence must describe the incident, never the act of looking at it. **4. The agent investigated our test suite.** Our tests import `app.main` , which calls `setup_telemetry()` , which shipped every span pytest produced into the real SigNoz. A live run read deployment-marker counts of `retry_storm=7` vs `prompt_regression=3` when only `prompt_regression` had been injected — the `retry_storm` markers were pytest's — and misdiagnosed on that basis. `AGENT_K_DISABLE_OTLP_EXPORT` now suppresses export while still building all three providers. **5. A 200 from a single-page app is not a resolving link.** We built evidence links to `/deployments` , a route that does not exist in SigNoz. Our link checker passed anyway — SigNoz is an SPA and answers 200 for every path, so a status check can't tell a real route from a typo. A human clicking the link landed on nothing. Deployment markers are spans, so the honest destination was the traces explorer. (Our default base URL was wrong too: `3301` , SigNoz's older port, where Foundry serves on `8080` .) Agent K never holds the Docker socket. A separate `deployer` sidecar does, and it exposes exactly one mutating endpoint that takes **an empty body** — the caller names no image, no service, no command. A caller that cannot name an image cannot be tricked into deploying an attacker's image. ``` deployer: volumes: # The privilege boundary, in one line. Nothing else in this project gets it. - /var/run/docker.sock:/var/run/docker.sock ``` One rule in there took a failed rollback to learn: the command is `docker compose up -d --force-recreate` , never `restart` . `restart` brings the container back on its *current* image and reports success while changing nothing — the rollback appears to work and the incident continues. *The Agent K dashboard in SigNoz: policy verdicts, why actions were denied, and recovery-verified counts side by side.* The honest summary: the model is the least trustworthy component in the system, and designing around that made everything else simpler. All of this worked for our setup — one machine, one service, four failure modes we wrote ourselves. We haven't tested any of it against an incident we didn't author. **Code:** [github.com/SujalXplores/Agent-K](https://github.com/SujalXplores/Agent-K) · Built for [Agents of SigNoz](https://www.wemakedevs.org/hackathons/signoz), Track 01. Built with AI coding assistance (Claude Code), disclosed in the repo; the shipped agent itself runs on free-tier providers and local embeddings only.