We ran Agent K against 12 seeded production incidents. It named the correct root cause 5 times out of 12 — 3 out of 9 if you discard the runs we couldn't cleanly attribute. Over the same 12 runs it produced the correct rollback / no-rollback verdict 10 times, published zero claims without a SigNoz link behind them, and executed zero actions outside its policy gate.
That gap between "diagnosed correctly" and "behaved correctly" is the entire project.
Agent K was our entry for Agents of SigNoz (WeMakeDevs × SigNoz), Track 01. The setup: a FastAPI RAG support service — 72 synthetic help-centre docs in Postgres + pgvector, local sentence-transformers
embeddings, free-tier models behind one OpenAI-compatible client (Groq by default; Cerebras, Gemini and OpenRouter swap in via one env var) — instrumented with OpenTelemetry, shipping traces, metrics and logs to SigNoz over OTLP. We installed SigNoz with Foundry, so casting.yaml
/casting.yaml.lock
are committed and the stack is reproducible; pointing the same exporter at SigNoz Cloud later took no code change.
When an alert fires, Agent K investigates through the SigNoz MCP server, publishes only claims carrying a resolvable SigNoz link, and rolls the service back only if a piece of plain Python says it may.
We started from an assumption we never tried to engineer away: the model will be wrong. So correctness can't be the safety mechanism. The safety has to be code.
Everything hangs off an in-process flag store with four seeded failures, and one deliberate asymmetry:
FLAG_NAMES: tuple[str, ...] = (
"prompt_regression", "retry_storm", "retrieval_latency", "db_pool_exhaustion",
)
DEPLOYMENT_CLASS_FLAGS: set[str] = {"prompt_regression", "retry_storm"}
SigNoz has no native deployment-marker API (we chased SigNoz/signoz#6162, closed unshipped), so we emit our own deployment.marker
OTel span through the pipeline that already exists. Two scenarios emit one; two don't. That present/absent asymmetry is the queryable signal — the absence of a marker is positive evidence that a rollback would not help.
One POST /ask in SigNoz: the rag.retrieval, rag.prompt_construction and chat spans, with the free SQLAlchemy DB span nested under retrieval.
Six deterministic checks, all of which must pass:
ACTION_ALLOWLIST: tuple[str, ...] = ("rollback",) # exactly one entry
SANDBOX_SERVICES = frozenset({"agent-k-rag-service"})
BURN_RATE_THRESHOLD = 1.0
CONFIDENCE_THRESHOLD = 0.70
COOLDOWN_SECONDS = 600.0
slo_breach
, allowlist
, cooldown
, confidence
, deployment_related
, sandbox_scope
. Every unknown fails closed — an alert we can't read a burn rate from parses to 0.0
, which fails the SLO check and denies the action.
The "no LLM" part isn't a comment we trust ourselves to honour. One test walks the module's import graph with ast
and fails the build if app.llm
ever appears in it; another monkeypatches llm.generate
to raise on any call and then runs a full evaluation, which catches an indirect call a grep would miss. Both live in the 219-test suite.
This is the part worth reading twice.
The deployment_related
check needs to know which scenario it's looking at, and the cheapest source was the model's claim text — our system prompt forces it to start its answer with one of the four scenario names. Fine, until the first 12-run eval printed this: two db_pool_exhaustion
incidents narrated as retry_storm
, each approving a rollback that could not possibly have fixed a database pool problem.
The gate wasn't broken. It was doing exactly what we wrote — and what we'd written was "the model may authorise a rollback by writing the word retry_storm in a sentence." Every safety property we were claiming had a hole in it shaped like a string match.
The fix was to make the check require corroboration from telemetry, not narration:
observed = frozenset(deployment_markers or ())
is_deployment_class = incident_type in DEPLOYMENT_CLASS_FLAGS
marker_seen = incident_type in observed
deployment_ok = is_deployment_class and marker_seen
And deployment_markers
is only ever populated from a tool result, never from a claim:
if ev_type == "deployment":
inv.deployment_markers_seen.update(
name for name in FLAG_NAMES if f'"{name}"' in content_text
)
We also made the ordering load-bearing — an investigation that stopped before the marker query could never approve anything — and asserted it at import rather than leaving it in a comment:
assert _DEPLOYMENT_QUERY_INDEX < MIN_EVIDENCE_ITERATIONS, (
"the deployment-marker query must fall within MIN_EVIDENCE_ITERATIONS, or the "
"policy gate can never corroborate a deployment-class claim"
)
After that fix, both bad approvals disappeared. The agent still misdiagnoses — it called retrieval_latency
"retry_storm" in all three of those runs — but a misdiagnosis now produces a denial, not a rollback.
The incident report's policy decision: five checks pass, deployment_related fails, and the action is denied with a next step for the human.
1. SigNoz's OTLP receivers don't bind until you finish first-run setup. A freshly cast stack looks healthy — every container up, curl localhost:8080
returns 200 — while curl localhost:4318/v1/traces
gives connection reset by peer, which reads exactly like a broken exporter. The collector is OpAMP-managed and can't register an agent without an org, so the backend loops on cannot create agent without orgId
. Check it directly:
curl -s http://localhost:8080/api/v1/version # "setupCompleted":false
Register an org (UI or POST /api/v1/register
) and the receivers come up. We lost real time believing app/telemetry.py
was wrong. It wasn't.
2. We invented MCP tool names. We'd written query_traces
, query_logs
, query_metrics
against a mental model of the API. The real inventory in signoz-mcp-server v0.9.0 is signoz_search_traces
, signoz_search_logs
, signoz_aggregate_traces
. Every evidence query failed, stripping every claim and escalating every investigation — a total failure that looked like an agent problem. Two more traps followed: no Windows binary ships in the releases (we cross-compiled in WSL), and SigNoz v0.134 rejects JWTs sent via SIGNOZ-API-KEY
, wanting Authorization: Bearer
— with a token that expires every 30 minutes, which is a delightful thing to discover mid-demo.
3. The free tier is a real design constraint. A 20-row trace search measured 8,424 tokens and came back HTTP 413 before a hypothesis existed. SigNoz returns every possible span attribute per row and most are null for this app, so we strip nulls rather than truncate — same signal, an order of magnitude fewer tokens. Then the model began diagnosing from the query engine's own statistics: "db_pool_exhaustion is likely due to an extremely high number of rows scanned (1018) and bytes scanned (10434)"
. Those are the stats for Agent K's own query. Hence:
_ENGINE_NOISE_KEYS = frozenset({
"meta", "rowsScanned", "bytesScanned", "durationMs", "stepIntervals",
"nextCursor", "queryName", "columnType", "aggregationIndex", "signal",
"fieldContext", "fieldDataType", "id", "warnings",
})
Evidence must describe the incident, never the act of looking at it.
4. The agent investigated our test suite. Our tests import app.main
, which calls setup_telemetry()
, which shipped every span pytest produced into the real SigNoz. A live run read deployment-marker counts of retry_storm=7
vs prompt_regression=3
when only prompt_regression
had been injected — the retry_storm
markers were pytest's — and misdiagnosed on that basis. AGENT_K_DISABLE_OTLP_EXPORT
now suppresses export while still building all three providers.
5. A 200 from a single-page app is not a resolving link. We built evidence links to /deployments
, a route that does not exist in SigNoz. Our link checker passed anyway — SigNoz is an SPA and answers 200 for every path, so a status check can't tell a real route from a typo. A human clicking the link landed on nothing. Deployment markers are spans, so the honest destination was the traces explorer. (Our default base URL was wrong too: 3301
, SigNoz's older port, where Foundry serves on 8080
.)
Agent K never holds the Docker socket. A separate deployer
sidecar does, and it exposes exactly one mutating endpoint that takes an empty body — the caller names no image, no service, no command. A caller that cannot name an image cannot be tricked into deploying an attacker's image.
deployer:
volumes:
- /var/run/docker.sock:/var/run/docker.sock
One rule in there took a failed rollback to learn: the command is docker compose up -d --force-recreate
, never restart
. restart
brings the container back on its current image and reports success while changing nothing — the rollback appears to work and the incident continues.
The Agent K dashboard in SigNoz: policy verdicts, why actions were denied, and recovery-verified counts side by side.
The honest summary: the model is the least trustworthy component in the system, and designing around that made everything else simpler. All of this worked for our setup — one machine, one service, four failure modes we wrote ourselves. We haven't tested any of it against an incident we didn't author.
Code: github.com/SujalXplores/Agent-K · Built for Agents of SigNoz, Track 01. Built with AI coding assistance (Claude Code), disclosed in the repo; the shipped agent itself runs on free-tier providers and local embeddings only.