Your production traffic is your test suite.
Halley records your production agent runs as bit-fidelity cassettes, turns any run into a permanent regression test with one click, and replays your whole fixture library in CI at zero LLM cost, then bisects to the exact commit that broke it.
halley ci
(green) → prompt regression →halley ci
(red) →halley bisect
names the commit. ~15 s, $0, zero live calls.
Self-hosted. OpenTelemetry-native. Built for the era where your model keeps changing under you.
Every team shipping LLM agents hits the same three walls within a month of production:
You can't test cheaply. Dev and CI traffic hammering live LLM APIs taxes every iteration.You can't reproduce bugs."The agent hallucinated Tuesday." Tuesday is gone, and the model behind it has shifted since.** You can't tell when you got worse.**A one-word prompt tweak or a provider's quiet model upgrade degrades behavior with no stack trace, and teams lose real money before anyone notices.
Observability tools (Langfuse, Laminar, LangSmith, Phoenix, Helicone) show you the traces. None close the loop from "production run happened" back into "regression test our CI catches next time." Halley closes that loop.
Pre-alpha, but the core loop works end to end today.
Shipped: OTLP ingest across 5 trace dialects · dashboard (runs, spans, cost) · promote a run into a fixture · structural / schema / metric invariants · halley ci
deterministic $0 replay · halley bisect
· GitHub Action · Python record/replay shim.
Planned, not built: semantic (LLM-as-judge) invariants (inferred and editable, but the runner is a stub, off by default) · TypeScript recorder shim · GitHub App fixture push (local repo path today) · Kubernetes Helm (Docker Compose today).
What you get depends on how you capture, and Halley is honest about it:
Tier 1 (observability, zero Halley code): point any OTLP instrumentation at the ingester and get the full dashboard plus automatic invariant inference. Bodies are reconstructed fromgen_ai.*
events, so they arenot byte-faithful and can't drive bit-exact replay.Tier 2 (bit-fidelity replay, one-line client wrap): add the recorder shim and Halley captures the full raw request/response JSON, sohalley ci
replays exact responses at**$0** andhalley bisect
is deterministic. This is the loop in the GIF above.
Both write the same fixture format. Details: docs/fixture-format.md (D53).
git clone https://github.com/A-Wattamwar/halley
cd halley
cp .env.example .env
docker compose up
docker compose up
brings up the ingester, dashboard, databases, and the code-only worker (promote a run → fixture, edit invariants). The .env
carries dev-only defaults; copy it once and the stack runs with no further setup. Point any OTLP-instrumented app at the ingester and real traces start flowing.
New here? Read docs/SCENARIO.md first: a concrete real-world story of the loop. Then:
: dashboard + runner + terminal, the full record → promote → CI → bisect loop.docs/running-the-loop.md
: the locked fixture v1 contract and the two capture tiers.docs/fixture-format.md
: system design and data model.docs/ARCHITECTURE.md
: what's shipped and what's deferred.docs/ROADMAP.md
- Full index:
.
docs/
Quickstarts for your stack are below.
The dashboard drives the whole loop: promote a run, edit invariants, Run CI, Run bisect.
The two actions that re-run your code, halley ci
and halley bisect
, execute on a lightweight runner on your machine (the worker, run on the host), not in a generic server container. Replaying your agent needs your repo, your venv, and your keys, the same reason git bisect
, GitHub self-hosted runners, and Buildkite agents all execute where the code lives. The dashboard enqueues and displays; the runner executes and streams results back.
Prefer the terminal? Every dashboard action shows the exact halley
command to copy, so the runner is never a hidden requirement. When no runner is connected, the buttons switch to Copy command and jobs resolve to an honest needs_runner
state, never a fake spinner.
Full model, including which worker runs what, host vs. in-network ports, and starting a host runner: docs/running-the-loop.md.
Halley ships a GitHub Action ( .github/workflows/halley-ci.yml) that replays the fixture library in
pure mode ($0, no live calls) on every pull request, publishes JUnit results as a PR check, and on failure posts a comment with the
halley diff
and halley bisect
commands to investigate. A second workflow () builds and tests the codebase and permanently guards the D22 canonical-hash contract with a Python↔Rust parity check.
.github/workflows/ci.yml
Halley is an OpenTelemetry-native observability backend with one hero capability wrapped around it.
Every production agent run becomes a cassette: the exact sequence of LLM calls, tool inputs, tool outputs, timing, and intermediate state. With the Tier-2 recorder shim, this is bit-fidelity capture of raw request and response bodies, not best-effort reconstruction.
One click on any run in the dashboard turns it into a permanent fixture in your repo's test library. Halley automatically infers invariants from the run:
Structural: which tools got called, in what order, how many times.** Schema**: the shape of each LLM output and tool payload.** Metric**: latency and cost bounds.** Semantic***(planned, not yet shipped)*: optional LLM-as-judge for "is the new output equivalent to the recorded one." The runner is a stub and ships off by default. See the deferrals in.docs/ROADMAP.md
You edit, tighten, or remove any inferred invariant before it lands in the repo.
halley ci
replays your entire fixture library against the current code. Zero live LLM calls when the cassette matches. When a prompt changes, Halley runs in hybrid mode: tool responses stay cached, only the drifted LLM call goes live. A full failing run shows exactly which invariant broke on which fixture.
When a regression fires, halley bisect
binary-searches recent commits and points at the change that broke the invariant: prompt diff, model version bump, framework upgrade, new tool version. The error message names the commit.
Every fixture is a reproducible record of what the agent did and why. For regulated industries that need to reproduce an agent decision on demand, the cassette is the audit trail.
Teams shipping LLM agents to production who are tired of rediscovering the same customer bug, deploying prompt changes based on three hand-picked examples, paying for the same test inputs repeatedly, and not knowing whether today's model still handles yesterday's edge cases.
Regulated industries (finance, healthcare, legal) that need agent decisions to be reproducible on demand and currently cannot guarantee that.
Open-source maintainers of agent frameworks who want CI that proves their changes do not regress real user workflows.
Halley is compatible with any OTLP-emitting instrumentation, including OpenLLMetry, OpenInference, OpenTelemetry GenAI semantic conventions, Vercel AI SDK, Pydantic AI, and raw provider SDK auto-instrumentation. If your app already exports OTLP, you are most of the way to Tier 1.
| If you want... | Use... |
|---|---|
| Browser-agent replay and session recording | |
LangfuseSelf-hosted backend that turns production runs into deterministic regression tests and catches prompt and model regressions in CI for free****Halley Halley is not a Langfuse or Laminar replacement. It is the regression-testing loop those platforms do not ship. Use Halley because you want cheap, deterministic CI against real production scenarios, not because you want another dashboard.
flowchart TD
app["Your AI app<br/>OpenLLMetry · OpenInference · Vercel AI SDK · OTEL GenAI · halley-raw"]
app -->|"OTLP/gRPC :4317 · OTLP/HTTP :4318 · POST /v1/spans/json"| ingester
subgraph ingester["Rust Ingester"]
normalizer["Normalizer<br/>halley-raw · openllmetry · openinference · vercel-ai · otel-genai"]
end
ingester -->|CanonicalSpan| redis["Redis Streams<br/>halley:spans"]
redis --> writer["Writer Task<br/>same binary"]
writer --> clickhouse[("ClickHouse<br/>observations · bodies · pricing")]
writer --> postgres[("Postgres<br/>auth · projects · API keys · jobs")]
clickhouse --> dashboard["Next.js Dashboard"]
postgres --> dashboard
dashboard -->|promote| fixtures["halley/fixtures/<br/>in your repo"]
fixtures --> ci["halley ci<br/>replay + bisect"]
Full system design in docs/ARCHITECTURE.md.
Pick the quickstart that matches your stack. Each is under 150 lines: prerequisites, install, setup snippet, verification SQL, and a link to a fully working example app under examples/
.
| Stack | Quickstart | source_dialect |
|---|---|---|
| Python + OpenAI (Traceloop / OpenLLMetry) | ||
otel-genai
quickstart-typescript.mdopeninference
quickstart-vercel.mdvercel-ai
Once you have fixtures, docs/running-the-loop.md walks the full record → promote → CI → bisect loop across the dashboard, runner, and terminal.
Halley normalizes spans from five dialects into a single canonical schema. No code changes required. Point your existing OTLP exporter at the ingester.
| Dialect | Detection | Status | Adapter |
|---|---|---|---|
| halley-raw | source_dialect = "halley-raw" field |
||
| ✅ Stable | |||
traceloop.*
attributeopenllmetry.rsopeninference.span.kind
or llm.model_name
openinference.rsai.operationId
or ai.model.*
vercel_ai.rsgen_ai.system
or gen_ai.provider.name
(fallback)otel_genai.rsDetection runs in priority order: halley-raw → OpenLLMetry → OpenInference → Vercel AI → OTEL GenAI. Unknown attributes from any dialect are preserved verbatim in the attributes
map and never dropped.
Note on OpenLLMetry / Traceloop:traceloop-sdk >= 0.55.0
(released 2026-03-29) migrated to pure OTEL GenAI semconv and no longer emitstraceloop.*
attributes. Traffic from modern Traceloop versions routes through theotel-genai
adapter. Theopenllmetry
adapter handles users ontraceloop-sdk < 0.55
(legacytraceloop.*
namespace). See[.]docs/research/openllmetry-2026-migration.md
| Layer | Choice |
|---|---|
| Ingester | Rust (tokio , axum , tonic , prost ) |
| Queue and buffer | Redis Streams |
| Hot storage | ClickHouse (spans and contents) |
| Warm storage | Postgres (auth, projects, API keys, invariant definitions) |
| Fixture format | Portable JSON and content blobs under halley/fixtures/ in your repo |
| CI harness | halley CLI (Rust) plus GitHub Action |
| Dashboard | Next.js 14 (App Router, Server Components) + Tailwind + shadcn/ui |
| Recorder shim | Python (sdk-py/ ), wraps the provider client for Tier-2 bit-fidelity capture; TypeScript shim planned |
| Protocol | OTLP (gRPC and HTTP) aligned with OpenTelemetry GenAI semantic conventions |
| Worker | Node.js, BullMQ jobs for invariant inference, fixture write, CI replay, and bisect |
| Orchestration | Docker Compose (local); Kubernetes Helm planned post-launch |
Promote a run, edit invariants, then run CI and bisect against the real repo, with a runner-status pill and copy-paste terminal commands for every action.
A Run CI result: 19/19 invariants passed, $0 pure-mode replay.
A bisect result: the offending commit, named.
More screenshots (runs list, span inspector, runner states, API keys) are in docs/screenshots/.
Single-node HTTP ingest load test (Phase 2, Week 4).
| Metric | Result |
|---|---|
| Achieved sustained RPS | |
| 4,792 spans/sec | |
| p50 latency | 1.69 ms |
| p95 latency | 113.93 ms |
| p99 latency | 185.15 ms |
| Error rate | 0.00% |
| Test duration | 5 minutes |
| Total spans ingested | 1,438,636 |
| ClickHouse rows written | 1,438,636 (0 data loss) |
Hardware: Apple M2, 8 GB RAM, Docker Desktop (all services co-located).
Bottleneck: The ingester receiver is not the bottleneck. The 5-second sanity check achieved ~9K RPS with 5 VUs. At 5K RPS sustained, the ClickHouse writer becomes the constraint: batch inserts at 100ms intervals with a single writer task limit throughput to ~4.8K spans/sec. The Redis stream absorbed the burst (peak lag ~1.48M entries) and the writer drained all entries with 0 data loss after the test ended. p99 latency exceeded the 50ms target because the Redis XADD
call occasionally queues behind the writer's batch flush.
Reproduce:
docker compose up -d && make ready
make load-test
MIT. See LICENSE.
Built by Ayush Wattamwar and Hridaya Dande. We built this together.
Named after Edmond Halley, who turned a noisy archive of past observations into a reliable prediction of the future. Which is what this tool does for agents.