AI Agent Regression Testing: Production Traces to CI

Julien Wuthrich released Tracely, an MIT-licensed open-source project that turns failing production agent traces into deterministic CI regression tests by recording tool and model outputs as replayable fixtures. The tool packages a five-stage loop — OTLP trace ingest, structural and LLM-judge grading, failure clustering, promotion of a trace to a hermetic test case, and a GitHub Action gate that blocks merges reintroducing known failures — built on FastAPI, ClickHouse, Postgres with pgvector, Redis and MinIO. Wuthrich notes that existing tools like Langfuse, LangSmith and Promptfoo cover parts of the loop but none removes the main cost: routing every tool and model call through a recordable seam.

TL;DR: Trace-based regression testing turns a failing production agent trace into a CI test case. The trace's recorded tool and model outputs become fixtures, so the test replays offline and deterministically on every pull request. Tracely is an MIT-licensed open-source project that packages this loop: OTLP ingest, graders, failure clustering, promotion of a trace to a case, and a GitHub Action gate. Langfuse, LangSmith and Promptfoo cover parts of the same loop. None of them removes the main engineering cost, which is routing every tool and model call through a seam you can record and replay. tracely-gate GitHub Action in one self-hosted stack. Agent failures rarely show up in the test inputs a team writes on day one. They come from real conversations: a user phrases a refund request the planner did not expect, a tool returns an empty list instead of an error, or a model upgrade changes how often the agent calls a search tool. By the time someone sees the failure, it lives in a trace a nested record of every prompt, model response, tool call and result in one run . Teams usually handle this in one of two ways, and both leak: Trace-based regression testing closes that gap. The trace is the test: its exact inputs, the exact tool outputs the agent saw, and the model responses in recorded order. A grader decides pass or fail. The suite runs on every pull request and blocks merges that bring a known failure back. This is the same discipline as "write a failing test before you fix the bug" in ordinary software. The difference is that agents are non-deterministic and depend heavily on external services, so the test needs recorded fixtures before it can be repeated. The loop has five stages. Every implementation, whether bought or built, needs all five. | Stage | What happens | What breaks if it is missing | |---|---|---| | Observe | Every production run emits a trace OpenTelemetry gen ai. or OpenInference attributes indexed by agent, conversation, turn and step | You cannot reproduce a failure you never recorded | | Grade | Structural checks schema, tool-call order, no model needed and LLM judges score runs online | Failures are found by users, not by the system | | Triage | Failing runs are clustered by structural signature and semantic similarity | 31 broken runs become 31 tickets instead of one issue with a count | | Promote | A representative failing trace becomes a hermetic case with its fixtures recorded | The "test" re-calls live services and drifts | | Gate | CI replays the suite against the change and fails the build on regressions | The fix ships, and the next refactor quietly reverts it | The Promote stage is where most homegrown setups stop. Copying a trace's input into a dataset is easy. Copying the world the agent saw tool results, retrieval hits, intermediate model outputs and replaying it in order is the actual work. Tracely https://github.com/jwuthri-cloud/Tracely-ai is an open-source project by Julien Wuthrich that describes itself as "trace-native CI/CD for AI agents." It is MIT-licensed, has about 1.2k stars at the time of writing, and is still early a few dozen commits . Its stack is a Python backend FastAPI, Celery, uv and a Next.js 15 UI. It stores traces in ClickHouse, the case registry in Postgres with pgvector, the queue in Redis and blobs in MinIO/S3. A local demo starts with docker compose --profile demo up -d --build --wait . Any OpenTelemetry exporter can post to POST {endpoint}/v1/traces with a Bearer ingest key. Tracely reads both gen ai. semantic-convention attributes and OpenInference attributes, so existing instrumentation keeps working. The Python SDK tracelyai-sdk adds auto-instrumentation for OpenAI, Anthropic, Google, Mistral and LangChain: python import tracely sdk as tracely tracely.init endpoint="http://localhost:8000", api key="tracely dev key", service name="support-agent", env="prod", prod | staging | ci | dev — the gating axis instrument="auto", with tracely.trace agent="support-agent", conversation="conv-1", user="u 42" : client.chat.completions.create model="gpt-4o", messages= {"role": "user", "content": "Where is order ORD-4471?"} , The env field matters more than it looks. It separates production traces the source of failures from CI traces the ones the gate grades , so the same pipeline can serve both. Evaluators come in two kinds. Structural checks need no model. LLM judges run at conversation, run or span level and can output a score, boolean, number, text or JSON-schema result. Failing runs are then clustered "structurally and semantically" with a structural signature plus embedding similarity. In the project's words, "31 broken runs become one issue with a count." This is the most important idea in the project. Promoting a failing trace bundles "recorded input, tool and LLM outputs" as fixtures. The case is trusted only if it "must fail on the old code and pass on the fix." A regression test that passes on the buggy code is a test that cannot fail, which is the most common way evaluation suites lie to their owners. The contract makes that check part of the workflow instead of leaving it to discipline. Multi-turn behaviour uses scenarios instead of single cases: either a scripted conversation, or an adversarial goal that a red-team model improvises against. Replay runs offline against the recorded fixtures, with no API keys and no model spend. tracely gate exits non-zero, posts a commit status and upserts a PR comment. The bundled action has two modes. Tracely can call your agent through an entrypoint tracely replay planner --entrypoint my pkg.agent:run , or --cmd "node run.js" for non-Python agents , or it can grade the CI traces your own test job emitted: - uses: jwuthri-cloud/Tracely-ai/.github/actions/tracely-gate@master with: mode: gate agent: planner api: https://tracely.your-co.dev key: ${{ secrets.TRACELY KEY }} Pin that @master reference to a commit SHA before you rely on it. A gate that can change underneath you is not a gate. The README states it plainly: hermetic replay "requires your agent to route tool/model calls through the SDK's call tool / call llm seam." Auto-instrumentation is enough to record a trace. It is not enough to replay one, because replay means intercepting the call and returning the recorded value instead of making it. If your agent calls requests.get from inside a tool, or a framework makes model calls you cannot reach, those calls go live during replay. Other caveats from the project itself: judges and failure analysis degrade without an LLM key, the default AUTH MODE=dev with tracely dev key is open access and only suitable for local use, and the Celery worker does not hot-reload. None of the established tools is called "trace-native CI," but each covers some of the stages. The question is which stages you get for free and which you build. | Stage | Tracely | Langfuse | LangSmith | Promptfoo | |---|---|---|---|---| | Production tracing | Yes OTLP + SDK | Yes, core product | Yes, core product | No test runs only | | Online grading | Structural + LLM judges | Online evaluators on sampled traffic | Online evaluators | No | | Failure clustering | Built in | Manual filtering and annotation | Manual filtering and annotation | No | | Trace → test | Promote to hermetic case | Add trace to dataset | Add run to dataset | Copy input into YAML test case | | Recorded tool fixtures | Yes, bundled on promote | Inputs/expected outputs only | Inputs/reference outputs only | No, you write mocks | | CI gate | tracely-gate action, PR comment | Experiments via SDK in your CI job | Evaluations via SDK in your CI job | promptfoo eval + GitHub Action | | Offline / no model spend | Yes, by design | No, experiments call the model | No, evaluations call the model | No, unless you mock providers | | Maturity | Early, small community | Mature, MIT core | Mature, commercial | Mature, MIT | A few clarifications on that table. Langfuse and LangSmith datasets store inputs and expected outputs, not the world. Adding a trace to a Langfuse dataset https://langfuse.com/docs/evaluation/dataset-runs/datasets or a LangSmith dataset https://docs.langchain.com/langsmith/evaluation captures what went in and, optionally, what should come out. When you run an experiment, your application runs again, live. That is the right design for comparing a new prompt or model against a fixed set of inputs: you want the new model's real behaviour. It is the wrong design for a regression test of agent logic, because the tools return today's data instead of the data that triggered the bug. For the trade-offs between these two platforms, see our LangSmith vs Langfuse vs Phoenix comparison https://dev.to/articles/LangSmith-vs-Langfuse-vs-Phoenix-LLM-Agent-Observability . Promptfoo is the strongest CI gate and the weakest trace source. Its GitHub Action https://www.promptfoo.dev/docs/integrations/github-action/ and assertion language are mature, and it runs red-team suites no trace tool matches. But it never sees production. Every case is something a human typed, so the "trace to test" step is copy-paste. Our Langfuse vs Promptfoo comparison https://dev.to/articles/langfuse-vs-promptfoo covers why many teams pair the two. Tracely's real differentiator is the fixture bundle plus the fail-to-pass contract. The tracing and judging overlap heavily with Langfuse. Indeed, the project describes parts of its design as mirroring Langfuse's. What is new is treating promotion as a test-engineering operation with a validity check, not as a "save to dataset" button. If you already run Langfuse or LangSmith and do not want another stack, you can build the missing stages yourself. The core is a single seam that every tool and model call goes through. This is a generic sketch of the pattern, not any vendor's API: python import json, hashlib from contextvars import ContextVar mode = ContextVar "mode", default="live" live | record | replay tape = ContextVar "tape", default=None list of recorded calls def key kind, name, args : blob = json.dumps kind, name, args , sort keys=True, default=str return hashlib.sha256 blob.encode .hexdigest :16 def seam kind, name, fn, args : """Every tool call and LLM call goes through here.""" mode, tape = mode.get , tape.get if mode == "replay": rec = tape.pop 0 strict order, not lookup if rec "key" = key kind, name, args : raise AssertionError f"divergence at {kind}:{name}" return rec "output" out = fn args if mode == "record": tape.append {"key": key kind, name, args , "kind": kind, "name": name, "output": out} return out Three design choices in that sketch decide whether the tests are useful. refund with the original order ID", "did not call cancel subscription " and use an LLM judge only where structure cannot express the requirement. Then write the fail-to-pass check as an actual test step: run the case against the parent commit and require a failure, run it against the fix and require a pass. Without that step you will end up with cases that were promoted from a trace but never reproduced the bug, and they pass forever. It is not a replacement for every kind of evaluation. The approach pays off when you have steady production traffic, recurring failure modes, and orchestration code that changes often. That is the profile of most customer-facing tool-calling agents. A sequence that keeps the investment proportional: gen ai. attributes AI agent regression testing checks that a change to an agent's code, prompts or model does not bring back a failure that was already fixed. The strongest form builds each test from a real failing trace and replays it on every pull request. Hermetic replay runs an agent test with recorded tool and model responses instead of live calls. The test needs no network access, no API keys and no token spend, and it gives the same result on every run, which makes it usable as a CI gate. A fail-to-pass contract is a rule that a regression case must fail on the code that had the bug and pass on the code that fixes it. It proves the test actually reproduces the failure, instead of being a test that cannot fail. Not yet for most teams. Tracely overlaps with their tracing and judging but is an early project with a small community. Its distinctive parts are fixture-bundled promotion, failure clustering and an offline CI gate, which you can also build on top of an existing platform. Promptfoo can gate CI on agent outputs and run red-team suites, but it does not ingest production traces. You copy the trace's input into a test case by hand and write your own mocks for tool responses. No, but it keeps your options open. Tracely, Langfuse, LangSmith and Phoenix all accept OpenTelemetry traces, so instrumenting once with gen ai. attributes lets you change backends without re-instrumenting the agent. Originally published at fp8.co https://fp8.co/articles/AI-Agent-Regression-Testing-Production-Traces-to-CI . Subscribe for weekly AI engineering analysis at fp8.co/newsletters https://fp8.co/newsletters .