Agent Detective – find which agent broke your multi-agent pipeline

Agent Detective, a new open-source evaluation framework for multi-agent systems, identifies the first node where quality breaks in a pipeline by ingesting standard OpenTelemetry traces and rebuilding the execution graph. The tool, which is self-hosted and free, outputs a verdict with the origin, propagation path, and downstream cost, and can gate CI on an exit code without requiring a database, broker, or LLM by default.

Self-hosting stays free. to be first to know. Watch releases → Classifications still move between minor versions 0.2.0 reclassified what 0.1.0 called , so gate CI on the loop detected exit code— 0 clean, 1 incident, 2 could not run — never on an exact report type . An eval framework for multi-agent systems that names the culprit. Ingest standard OpenTelemetry traces, rebuild the execution graph, and find the first node where quality broke — the origin, the propagation path, and the downstream cost. OTEL-native: any OpenInference / OpenLLMetry instrumented agent works with no code change. pip install agent-detective detective analyze trace.json ── graph 3f2a91c8 content-pipeline FAILED · cut point · confidence 62% Origin — where quality broke translator Defects ● Contract breach — translator A carried input/output parameter was silently rewritten at translator. observation 100% · attribution 92% · channel deterministic | You want to… | Read | |---|---| | The guided tour — instrumenting, analyzing, CI, machine output | | | Connect your own agents attributes, adapters, SDK | | docs/architecture.md /Thomeras/agent detective/blob/main/docs/architecture.md docs/capabilities.md /Thomeras/agent detective/blob/main/docs/capabilities.md , docs/trace-requirements.md /Thomeras/agent detective/blob/main/docs/trace-requirements.md packages/detective cli/README.md /Thomeras/agent detective/blob/main/packages/detective cli/README.md php flowchart LR AG "your agent" -- "OTLP/HTTP JSON" -- CAP "detective capture :8900" AG -- "AGENT DETECTIVE TRACE FILE" -- F "run.json" CAP -- AN "detective analyze" F -- AN AN -- V "verdict · exit 1 on incident<br/ terminal · --json · --markdown" detective capture --once --out run.json receive a trace straight from your agent detective analyze run.json the verdict; exit 1 on an incident CI gate as-is detective analyze run.json --markdown findings brief for a coding agent detective doctor run.json is the trace even worth trusting? Already instrumented? Point the exporter at it: OTEL EXPORTER OTLP PROTOCOL=http/json OTEL EXPORTER OTLP ENDPOINT=http://127.0.0.1:8900 . Not yet? detective-sdk pure stdlib, zero dependencies emits the same spans: python from detective sdk import run with run "intel", task=user request as r: with r.step "write" as s: s.output = dossier markdown the work, not {"ok": true} s.cost usd=0.03, tokens in=8 000, tokens out=900, model="gpt-4o" No database, no broker, and by default no LLM: the deterministic evidence channel needs nothing but the trace. Set JUDGE BASE URL / JUDGE MODEL any OpenAI-compatible endpoint — OpenAI, OpenRouter, local Ollama; concrete configs in docs/usage.md /Thomeras/agent detective/blob/main/docs/usage.md 22-turning-on-the-per-node-judge to also turn on the per-node quality judge — without it, nodes report unscored , never silently "fine". examples/diamond eval.py /Thomeras/agent detective/blob/main/examples/diamond eval.py instruments a diamond — one extractor, two parallel writers, one editor — and gates on the verdict. Each writer declares the facts it was handed as a contract ; --inject makes the marketing branch silently rewrite the price: with r.branch "marketing writer", input=facts as s: s.contract price=facts "price" , availability=facts "availability" s.output = write marketing facts, inject --inject rewrites the price bash $ python examples/diamond eval.py --inject graph 0a68bb06: cut point — culprit: marketing writer exit code 1 The verdict names the branch that rewrote the fact — not the editor who merged it downstream — and verifies the breach actually shipped, all with no LLM: FAILED · cut point · confidence 95% Origin — where quality broke marketing writer ● Contract breach — marketing writer supporting: contract breach rule input contract:price supporting: breach propagated at terminal contract-propagation check on the deliverable Deterministic signals fail contract violation marketing writer price: $12/user/month → from $5/user/month Point the same script at the full stack and the graph appears in the web UI — culprit ring on marketing writer , propagation path into press editor : AGENT DETECTIVE ENDPOINT=http://localhost:8001 python examples/diamond eval.py --inject docker compose up --build infra + ingest + worker + api + web ./demo/run.sh happy-path demo: graph, no incident ./demo/inject fault.sh && ./demo/run.sh a cut point incident appears Web UI at :5173 , read API at :8000 , ingest at :8001 ; the bundled mock LLM judges, so no external API keys are needed. The self-hosted stack has no authentication yet.Every host binding is therefore on 127.0.0.1 by default — the UI and API, and Postgres, ClickHouse, Redis and MinIO with them. What sits behind those ports is your traces' payloads: agent inputs and outputs verbatim, which is exactly the material you would not want public. Reaching the stack from elsewhere is one deliberate knob, and only belongs behind something that authenticates: BIND=0.0.0.0 docker compose up --build Auth and multi-tenancy are what Cloud is being built for. php flowchart LR AG "your agents" -- "OTLP/HTTP" -- ING "ingest :8001" ING -- CH "ClickHouse<br/ raw spans" ING -- PG "Postgres<br/ graphs · runs · edges" ING -.- |"payload overflow"| MIO "MinIO" ING -- "Redis Streams" -- T1 "worker tier1<br/ cheap checks + terminal judge" T1 -- "flagged / sampled" -- T2 "worker tier2<br/ per-node scoring + judge" T2 -- BE "blame engine<br/ pure networkx " BE -- INC "incidents +<br/ blame reports" INC -- API "read API :8000" API -- UI "web UI :5173" The report separates where quality broke origin from where it surfaced manifestation , flags rubber-stamping verifiers, and reports capped, split confidence observation × attribution . What the trace did not capture, no analysis can manufacture — absent evidence renders unverified , never ok . Your agent's process needs only detective-sdk 25 KiB, zero dependencies : it emits the trace and never computes anything. The verdict happens wherever is installed — your laptop, CI, or the deployed stack — and that one install brings the whole analysis side with it agent-detective otel-mapper span mapping, agent-detective-worker pipeline, blame-engine verdicts, the CLI . detective-ci is a separate, opt-in install because it registers a pytest plugin. Details per distribution: docs/usage.md §4.4 /Thomeras/agent detective/blob/main/docs/usage.md 44-packages-and-distributions . packages/ blame engine/ pure, I/O-free blame analysis networkx only otel mapper/ OTLP span - AgentRun/Edge mapping Apache-2.0 detective cli/ the agent-detective pip distribution: local mode + CLI detective sdk/ zero-dependency instrumentation helpers detective ci/ deterministic golden replay + pytest plugin services/ ingest · worker tier1/tier2 + judge · read API db/ Alembic migrations docker/clickhouse/ ClickHouse init web/ React + Vite + cytoscape UI uv sync --all-packages --all-groups install the workspace ./scripts/test.sh every unit suite mirrors CI uv run pytest tests/e2e acceptance test needs a running stack for p in blame-engine otel-mapper detective-sdk detective-ci \ agent-detective-worker agent-detective; do uv build --package "$p" the pip distributions done Business Source License 1.1 see LICENSE /Thomeras/agent detective/blob/main/LICENSE ; packages/otel mapper is Apache-2.0.