Agent traces helped us see what went wrong, but surfacing the failures was almost always a manual trigger. Someone had to spot the anomaly, dig through spans, and then hand‑roll a fix. That delay is exactly where Selfship.ai lives.
It is an autonomous loop that reads every trace, turn, and multi‑turn conversation looking for four classes of drift:
- Did the user actually get what they asked for, or did the agent happily hallucinate a confirmation?
- Are specific tool calls failing repeatedly under the same user intent?
- Do users keep rephrasing the same question instead of getting a straight answer?
- Is the agent still taking the longest‑available path to a known outcome?
The core promise is simple: group failures by user intent, evaluate them, and ship fixes as pull requests. Once a fix lands, the system re‑evaluates the same traces to confirm the regression is gone. If it is not, the loop keeps nudging until someone merges something that actually sticks.
We just opened Selfship.ai up as a SaaS, and the onboarding is lighter than most observability tools because it leans on traces you are already emitting. You point it at a trace backend, it starts clustering, and within hours you get ranked issue lists tied to concrete user intents.
A few things we learned the hard way that the tool now bakes in:
-
Intent clustering beats error codes. Raw span errors are noisy; "user asked for X and got Y" is a signal you can act on.
-
PR‑driven fixes close the loop. Linking each detected issue to an actual diff means you stop triaging and start shipping.
-
Regression checks survive deploys. Post‑merge evaluation catches the fixes that looked good in staging but flop in the wild.
If you have an agentic product in production and you are still relying on human‑in‑the‑loop trace reviews, this is worth a look. Drop a trace ID and we will run it through the clustering engine free of charge. Next Built an app for photos your phone's gallery can't handle →