Why my alert triage workflow needed a CLI

A developer built a CLI wrapper around SigNoz's published OpenAPI spec to give an AI agent a stable interface for alert triage, after an initial attempt that pasted Slack dumps into an agent session proved slow and error-prone because the agent wrote extensive Python and made incorrect assumptions about request bodies and response shapes. The developer generated an Effect client from the OpenAPI spec, wrapped it in a CLI, and added a Pi agent skill instructing the agent to investigate root causes with the CLI and cite the commands behind every assertion, which anecdotally reduced hallucination. Two remaining issues — unreadable JSON output and 15-plus tool calls spent getting oriented per alert — were addressed with human-readable table output formats and updated skill instructions.

All writing https://powers.dev/writing Why my alert triage workflow needed a CLI I used AI to turn my notes into a draft and to edit the result. The experiences and ideas are my own. I wanted to hand an agent an alert from Slack and have it help me figure out what was wrong. We used SigNoz for observability. I set up a read-only API key in the environment so the agent could make authenticated requests to the SigNoz API while it investigated. My first attempt was pretty straightforward: copy a Slack dump into a new agent session and let it run. It worked It was also very slow and error-prone. The agent wrote a lot of Python. It made incorrect assumptions about request bodies and response shapes, then spent time working through the resulting errors. I wanted help investigating an alert, but a substantial part of the session went toward figuring out how to talk to the API. So I went looking for an OpenAPI spec. Fortunately, SigNoz published one. Verifiable evidence SigNoz also had an MCP server, which I considered. I use Pi as my agent harness, and using the server in my setup would have required an adapter. That was one reason to look elsewhere, but I also wanted a way to verify the data behind the agent's conclusions. If it found something interesting, I wanted a command I could run myself to see the same information directly from SigNoz. I didn't want to reconstruct an API request or dig through a pile of generated Python to check its work. A CLI seemed like a good fit. The agent could use it from a shell, and I could copy a command into my own terminal. I generated an Effect client from the OpenAPI spec and wrapped it in a CLI https://github.com/jpowersdev/signoz-cli . That gave the agent an interface built around the published API instead of having it improvise requests throughout each investigation. Then I made an agent skill for Pi to use during triage. The main instruction was simple: use the new CLI to investigate the root cause, and when making an assertion, include the commands that produced the evidence supporting it. The reports started with a prose section describing the problem, what caused it, and the supporting evidence. They ended with a numbered list of references. I could read the explanation, copy a referenced command, and run it myself. Before I added that requirement, triage reports often mixed useful data from SigNoz with guesses about what it meant. In one report, the agent saw a slow login span and speculated that our database connection pool was full, leaving requests waiting for a connection. The real cause turned out to be Better Auth's Sentinel plugin doing proof of work. It was slow on purpose. Once every assertion had to be explicitly justified, I noticed that the agent hallucinated much less, anecdotally at least. If it couldn't justify something, it didn't claim it. That was really nice. I still had to decide whether the evidence supported the explanation, but I could actually inspect that evidence. The report gave me somewhere concrete to start. Two remaining annoyances Once I started using it on actual alerts, two problems became obvious. First, you can reproduce the exact query, but that isn't especially helpful when the result is a massive JSON blob. I had made the data accessible without making it pleasant to read. Second, the agent would spend 15 or more tool calls just getting oriented. It would run every help command, then query random services, traces and metrics to see what was available. Each new alert came with another round of setup. I made three changes. Output I could read I added output formats, including human-readable tables, and updated the triage skill to request the human-readable format. Now, when I copied a command from the report, I got something I could actually read. It was a small change, but it made following the references much less tedious. The verification step needed to be convenient enough that I would actually do it. Instructions in one place I added signoz agent instructions . It brought together help for the commands and subcommands, along with guidance on when to use them. Instead of making the agent walk through the CLI one help invocation at a time, I could give it the guide in one command. Context from the actual instance Knowing how to use the tool wasn't enough. The agent also needed to know what it could query in our SigNoz instance. I added signoz agent context . It ran discovery queries in parallel and combined the results into a single overview of what was available. It also grouped related entries, including by prefix, to make the overview more compact. The two commands handled different parts of the setup: how to use the CLI, and what data was there to investigate. Finally, the workflow I wanted At that point, I was getting consistently useful root-cause reports in my own triage work. The agent spent less time figuring out its tools, and I got explanations with references I could follow and output I could read. I could give it an alert, read the investigation, and check the commands behind its findings. That was what I'd wanted from the beginning. For about three weeks, it was glorious. And then we switched to Grafana.