{"slug": "tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests", "title": "Tracely: Open-Source CI/CD That Turns Failed Agent Runs Into Regression Tests", "summary": "Tracely, an MIT-licensed open-source project created by GitHub user jwuthri-cloud on October 11, 2026, converts failed AI agent production runs into replayable regression tests that gate pull requests in CI, according to the project's README and documentation. The repository had 1,163 stars and 140 forks as of October 24, 2026, and its Python SDK, tracelyai-sdk, is on PyPI at version 0.0.1; traces arrive over plain OTLP or the SDK, and the README says CI replays use recorded fixtures offline with no API keys. The project is early-stage and the review found no independent benchmark of its claims, with the authors noting they did not deploy it in production.", "body_md": "Tracely is a new open-source project that tries to close the loop most agent observability tools leave open: when an agent breaks in production, the failing run becomes a regression test that blocks the next pull request that would ship the same bug. The repository by `jwuthri-cloud` was created on October 11, 2026 and had 1,163 stars and 140 forks on October 24, according to the [GitHub API](https://github.com/jwuthri-cloud/Tracely-ai?ref=explainx). It is MIT licensed and self-hostable.\n\nWe read the [README](https://github.com/jwuthri-cloud/Tracely-ai?ref=explainx), the project docs and the PyPI listing for the SDK. We did not deploy it in production, so everything below is what the project says it does, with our notes on what to verify yourself.\n\n## Quick answers\n\n| Question | Answer | \n|---|---|\n| What does it do? | Grades agent traces, clusters failures, freezes them into replayable cases, gates PRs in CI | \n| License | MIT | \n| Hosting | Self-host (Docker Compose, Railway template, Kubernetes manifests) | \n| How do traces get in? | Plain OTLP over HTTP, or the Python SDK `tracelyai-sdk` | \n| Do CI replays cost tokens? | README says no: recorded fixtures, offline, no API keys | \n| Maturity | Early: repo created October 11, 2026; SDK on PyPI at 0.0.1 | \n| Stars | 1,163 (about 13 days old) | \n\n## What problem is Tracely going after?\n\nThe README opens with a blunt critique: \"observability stops at the dashboard.\" You can see that an agent broke, but turning that into a durable test is manual work. Eval tools usually ask you to write a dataset by hand, which the project calls \"a guess about what might break.\"\n\nTracely's alternative is that the recorded production run is the test. It stores the exact input, the exact tool calls and the exact model responses, then replays them on every pull request. The README compares itself with dataset-first tools on five points: where tests come from, fidelity to production, cost to replay in CI, what happens on a regression, and how you find out.\n\nThat framing is the project's own marketing. It is a reasonable idea, but we found no independent benchmark of it. The same thinking shows up elsewhere: our piece on [AI evals for engineers and PMs](https://explainx.ai/blog/ai-evals-for-engineers-and-pms-explained-2026) explains why production-derived test cases beat invented ones, and [LangSmith's Jev](https://explainx.ai/blog/langsmith-jev-production-trace-scoring-2026) scores live production traces in a similar spirit.\n\n## The five-step loop\n\nThe README describes a pipeline of \"production trace, failure detection, regression test, CI gate, alert\". Each step has its own page in the app.\n\nAgent observability trace view illustrating how Tracely records each production run\n\n### 1. Observe\n\nTraces arrive over standard OTLP. Agent concepts such as `agent.id`, `conversation.id`, `turn` and `step` are promoted to indexed columns, so runs group into conversation threads rather than a flat pile of spans. Evaluators appear as columns on the trace table, and scores stream in over server-sent events as judges finish.\n\n### 2. Detect and triage\n\nOnline evaluators grade runs as they land. The README lists LLM-as-judge grading at conversation, run and span level, plus structural checks that need no model. Failures are then clustered, structurally and semantically, so that, in the README's example, 31 broken runs become one issue with a count. Each cluster can suggest the evaluator that would have caught it.\n\n### 3. Test\n\nOne click promotes a failing trace to a hermetic case: recorded input, with tool and LLM outputs bundled as fixtures. Each case carries a fail-to-pass contract. It must fail on the old code and pass on the fix, or the promotion is not trusted. For multi-turn behavior there are scenarios, either scripted conversations or an adversarial goal that a red-team model pursues against your agent's endpoint.\n\n### 4. Ship\n\nPull request merge gate where a failing regression case blocks the change in CI\n\nThe CI suite replays against recorded fixtures. The CLI offers `tracely simulate`, `replay` and `gate`, plus a GitHub Action. A failing `tracely gate` exits non-zero, posts a commit status and upserts a PR comment. The README stresses this is deterministic and offline: no API keys and no model spend.\n\n### 5. Alert\n\nAlert rules have a trigger (a gate fails, a live conversation fails a judge, a new failure mode appears, a rate crosses a threshold) and a flow drawn on a canvas: conditions, Slack, email, a webhook, an LLM step, or a Python expression. The README says an in-app assistant can draw the flow from a description.\n\nThere is also a judge calibration screen. You label judge verdicts against human review and see each judge's agreement rate, which helps you catch an over-flagging judge before it gates a release. That matters, because a CI gate driven by a noisy LLM judge is a good way to make a team ignore the gate.\n\n## A worked example of the loop\n\nImagine a support agent that looks up orders. A customer asks about an order, the agent calls the lookup tool with a malformed id, and the reply invents a delivery date. In Tracely terms, a judge flags that run, the cluster step groups it with similar runs into one issue, and you promote the worst example to a case. The case stores the customer message, the tool call and the recorded tool response. After you fix the id handling, the case must pass, and on the old code it must fail. From then on every pull request replays it, so a later prompt edit that reintroduces the bug turns the check red.\n\n## How to try it\n\nThe README quickstart needs Docker and Docker Compose. The demo profile starts the whole stack and seeds sample traces, clusters, cases and gates, so the UI is populated on first open:\n\n```\ngit clone https://github.com/jwuthri-cloud/Tracely-ai\ncd Tracely-ai\ndocker compose --profile demo up -d --build --wait\n```\n\nThe UI runs on port 3001 and the backend on 8000 by default. The stack includes ClickHouse, Postgres, Redis and MinIO, which is a lot of moving parts for a trial. The README also offers a Railway one-click template and an EKS and Kubernetes walkthrough, and points to a production-hardening runbook.\n\nTo send your own traces, install the SDK from [PyPI](https://pypi.org/project/tracelyai-sdk/?ref=explainx) with an extra for your provider, for example `pip install \"tracelyai-sdk[openai]\"`. The README shows initializing once with an endpoint, an ingest key, a service name and `instrument=\"auto\"`, which detects OpenAI, Anthropic, Google, Mistral and LangChain clients. You then wrap a run in a `tracely.trace(...)` context with an agent name, conversation id and user. The default dev ingest key in the demo is a local convenience only; change it for anything shared.\n\nThe API also exposes an MCP endpoint, so a coding agent can drive the workspace with an ordinary ingest key, and the repo ships a skill for coding agents. If you are new to those ideas, see our guide to [agent skills](https://explainx.ai/blog/what-are-agent-skills-complete-guide) and [what MCP is](https://explainx.ai/blog/what-is-mcp-model-context-protocol-guide).\n\n## What to check before you adopt it\n\nAgent working inside a fenced area, a reminder to sandbox what you let observe your traces\n\n**Data in traces.** Production traces contain real user inputs, tool outputs and model responses. Self-hosting keeps them in your infrastructure, but the grading judges call a model, and the README mentions a per-workspace OpenRouter key. Decide what leaves your network, and redact personal data before ingest.\n\n**Maturity.** The SDK is at 0.0.1 and the repo is under two weeks old, with 22 open issues when we checked. The README's near-term plan points to a PRD of next steps. Star counts show interest, not stability.\n\n**Fixture fidelity.** Replaying recorded tool and LLM outputs proves your agent code handles that exact transcript the same way. It does not prove the agent would choose the same action with a different model or prompt, since the model is mocked. Use it as a regression net, not as a replacement for live evals on prompt or model changes.\n\n**Judge quality.** Failure detection leans on LLM judges. Use the calibration screen and keep a human-labeled sample.\n\n**Operational weight.** Four datastores (ClickHouse, Postgres, Redis, MinIO) plus an API, a worker and a UI is real operational cost. If you only need tracing, a lighter tool such as the [LiteLLM Lens analyzer](https://explainx.ai/blog/litellm-lens-agent-trace-analysis-2026) or Perplexity's [Numbat](https://explainx.ai/blog/numbat-perplexity-ai-agent-observability-tool-2026) may be enough. The wider question of how to catch regressions in AI-generated patches is covered in [Trail of Bits' regression-hunt plugins](https://explainx.ai/blog/trail-of-bits-patching-bugs-regression-hunt-plugins-21-regressions-130-prs-patch-the-planet-2026).\n\n## Where it fits in the agent testing landscape\n\nAgent teams currently choose among three habits. Some watch dashboards and fix by hand. Some maintain curated eval datasets, such as the benchmark work in [LangChain's Jev agent evals](https://explainx.ai/blog/langchain-jev-agent-evals-benchmark-september-2026). Some are starting to treat production traces as the raw material for tests. Tracely is a bet on the third, packaged as a full product with a UI, alerts and CI integration rather than a library.\n\nIf the idea holds up, the practical effect is cultural: every incident leaves behind a test, so the same failure cannot silently return. That is exactly what unit tests did for ordinary software. The open question, which only real use can answer, is how many promoted cases stay stable as prompts and tools change, and how much human review the fail-to-pass step really needs.\n\nFor sandboxing the agents you test, see our coverage of [NVIDIA's OpenShell runtime](https://explainx.ai/blog/nvidia-openshell-github-hacker-news-409-points-sandbox-runtime-agents-telemetry-what-developers-say-2026).\n\n## Bottom line\n\nTracely is worth a look if you run agents in production and your current process for turning incidents into tests is \"someone remembers.\" Try the demo profile, send a trace from a non-critical agent, and measure how often the clustered issues and promoted cases match what you would have written by hand. Wait for a stable SDK release before betting a release process on it.\n\n## Related reading\n\nSources: the [Tracely GitHub repository](https://github.com/jwuthri-cloud/Tracely-ai?ref=explainx) and the [tracelyai-sdk PyPI page](https://pypi.org/project/tracelyai-sdk/?ref=explainx).\n\n*Details reflect the repository as of October 24, 2026 and may change.*", "url": "https://wpnews.pro/news/tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests", "canonical_source": "https://explainx.ai/blog/tracely-ai-open-source-trace-native-ci-cd-ai-agents-regression-tests-github-trending-2026?utm_source=rss&utm_medium=feed&utm_campaign=blog", "published_at": "2026-10-24 00:00:00+00:00", "updated_at": "2026-10-24 03:42:22.952715+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "developer-tools", "ai-safety"], "entities": ["Tracely", "jwuthri-cloud", "tracelyai-sdk", "PyPI", "GitHub", "LangSmith"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests", "markdown": "https://wpnews.pro/news/tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests.md", "text": "https://wpnews.pro/news/tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests.txt", "jsonld": "https://wpnews.pro/news/tracely-open-source-ci-cd-that-turns-failed-agent-runs-into-regression-tests.jsonld"}}