# Agent Detective – find which agent broke your multi-agent pipeline > Source: > Published: 2026-07-29 18:07:02+00:00 Self-hosting stays free. [to be first to know.]Watch releases → Classifications still move between minor versions ( [0.2.0 reclassified what 0.1.0 called]), so gate CI on the`loop_detected` exit code—`0` clean,`1` incident,`2` could not run — never on an exact`report_type` . **An eval framework for multi-agent systems that names the culprit.** Ingest standard OpenTelemetry traces, rebuild the execution graph, and find the first node where quality broke — the origin, the propagation path, and the downstream cost. OTEL-native: any OpenInference / OpenLLMetry instrumented agent works with no code change. ``` pip install agent-detective detective analyze trace.json ── graph 3f2a91c8 [content-pipeline] FAILED · cut_point · confidence 62% Origin — where quality broke translator Defects ● Contract breach — translator A carried input/output parameter was silently rewritten at translator. observation 100% · attribution 92% · channel deterministic ``` | You want to… | Read | |---|---| | The guided tour — instrumenting, analyzing, CI, machine output | | | Connect your own agents (attributes, adapters, SDK) | | [docs/architecture.md](/Thomeras/agent_detective/blob/main/docs/architecture.md)[docs/capabilities.md](/Thomeras/agent_detective/blob/main/docs/capabilities.md),[docs/trace-requirements.md](/Thomeras/agent_detective/blob/main/docs/trace-requirements.md)[packages/detective_cli/README.md](/Thomeras/agent_detective/blob/main/packages/detective_cli/README.md) ``` php flowchart LR AG["your agent"] -- "OTLP/HTTP JSON" --> CAP["detective capture :8900"] AG -- "AGENT_DETECTIVE_TRACE_FILE" --> F["run.json"] CAP --> AN["detective analyze"] F --> AN AN --> V["verdict · exit 1 on incident
terminal · --json · --markdown"] detective capture --once --out run.json # receive a trace straight from your agent detective analyze run.json # the verdict; exit 1 on an incident (CI gate as-is) detective analyze run.json --markdown # findings brief for a coding agent detective doctor run.json # is the trace even worth trusting? ``` Already instrumented? Point the exporter at it: `OTEL_EXPORTER_OTLP_PROTOCOL=http/json OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:8900` . Not yet? `detective-sdk` (pure stdlib, zero dependencies) emits the same spans: ``` python from detective_sdk import run with run("intel", task=user_request) as r: with r.step("write") as s: s.output = dossier_markdown # the work, not {"ok": true} s.cost(usd=0.03, tokens_in=8_000, tokens_out=900, model="gpt-4o") ``` No database, no broker, and by default no LLM: the deterministic evidence channel needs nothing but the trace. Set `JUDGE_BASE_URL` / `JUDGE_MODEL` (any OpenAI-compatible endpoint — OpenAI, OpenRouter, local Ollama; concrete configs in [docs/usage.md](/Thomeras/agent_detective/blob/main/docs/usage.md#22-turning-on-the-per-node-judge)) to also turn on the per-node quality judge — without it, nodes report *unscored*, never silently "fine". [examples/diamond_eval.py](/Thomeras/agent_detective/blob/main/examples/diamond_eval.py) instruments a diamond — one extractor, two parallel writers, one editor — and gates on the verdict. Each writer declares the facts it was handed as a **contract**; `--inject` makes the marketing branch silently rewrite the price: ``` with r.branch("marketing_writer", input=facts) as s: s.contract(price=facts["price"], availability=facts["availability"]) s.output = write_marketing(facts, inject) # --inject rewrites the price bash $ python examples/diamond_eval.py --inject graph 0a68bb06: cut_point — culprit: marketing_writer # exit code 1 ``` The verdict names the branch that rewrote the fact — not the editor who merged it downstream — and verifies the breach actually shipped, all with no LLM: ``` FAILED · cut_point · confidence 95% Origin — where quality broke marketing_writer ● Contract breach — marketing_writer supporting: contract_breach (rule input_contract:price) supporting: breach_propagated at terminal (contract-propagation check on the deliverable) Deterministic signals fail contract_violation (marketing_writer) price: $12/user/month → from $5/user/month ``` Point the same script at the full stack and the graph appears in the web UI — culprit ring on `marketing_writer` , propagation path into `press_editor` : ``` AGENT_DETECTIVE_ENDPOINT=http://localhost:8001 python examples/diamond_eval.py --inject docker compose up --build # infra + ingest + worker + api + web ./demo/run.sh # happy-path demo: graph, no incident ./demo/inject_fault.sh && ./demo/run.sh # a cut_point incident appears ``` Web UI at `:5173` , read API at `:8000` , ingest at `:8001` ; the bundled mock LLM judges, so no external API keys are needed. The self-hosted stack has no authentication yet.Every host binding is therefore on`127.0.0.1` by default — the UI and API, and Postgres, ClickHouse, Redis and MinIO with them. What sits behind those ports is your traces' payloads: agent inputs and outputs verbatim, which is exactly the material you would not want public. Reaching the stack from elsewhere is one deliberate knob, and only belongs behind something that authenticates: ``` BIND=0.0.0.0 docker compose up --build ``` Auth and multi-tenancy are what Cloud is being built for. ``` php flowchart LR AG["your agents"] -- "OTLP/HTTP" --> ING["ingest :8001"] ING --> CH[("ClickHouse
raw spans")] ING --> PG[("Postgres
graphs · runs · edges")] ING -.->|"payload overflow"| MIO[("MinIO")] ING -- "Redis Streams" --> T1["worker tier1
cheap checks + terminal judge"] T1 -- "flagged / sampled" --> T2["worker tier2
per-node scoring + judge"] T2 --> BE["blame engine
(pure networkx)"] BE --> INC[("incidents +
blame reports")] INC --> API["read API :8000"] API --> UI["web UI :5173"] ``` The report separates **where quality broke** (origin) from **where it surfaced** (manifestation), flags rubber-stamping verifiers, and reports capped, split confidence (observation × attribution). What the trace did not capture, no analysis can manufacture — absent evidence renders `unverified` , never `ok` . Your agent's process needs only ** detective-sdk** (25 KiB, zero dependencies): it emits the trace and never computes anything. The verdict happens wherever **is installed — your laptop, CI, or the deployed stack — and that one install brings the whole analysis side with it (** `agent-detective` `otel-mapper` span mapping, `agent-detective-worker` pipeline, `blame-engine` verdicts, the CLI). `detective-ci` is a separate, opt-in install because it registers a pytest plugin. Details per distribution: [docs/usage.md §4.4](/Thomeras/agent_detective/blob/main/docs/usage.md#44-packages-and-distributions). ``` packages/ blame_engine/ pure, I/O-free blame analysis (networkx only) otel_mapper/ OTLP span -> AgentRun/Edge mapping (Apache-2.0) detective_cli/ the `agent-detective` pip distribution: local mode + CLI detective_sdk/ zero-dependency instrumentation helpers detective_ci/ deterministic golden replay + pytest plugin services/ ingest · worker (tier1/tier2 + judge) · read API db/ Alembic migrations docker/clickhouse/ ClickHouse init web/ React + Vite + cytoscape UI uv sync --all-packages --all-groups # install the workspace ./scripts/test.sh # every unit suite (mirrors CI) uv run pytest tests/e2e # acceptance test (needs a running stack) for p in blame-engine otel-mapper detective-sdk detective-ci \ agent-detective-worker agent-detective; do uv build --package "$p" # the pip distributions done ``` Business Source License 1.1 (see [LICENSE](/Thomeras/agent_detective/blob/main/LICENSE)); `packages/otel_mapper` is Apache-2.0.