{"slug": "agent-detective-find-which-agent-broke-your-multi-agent-pipeline", "title": "Agent Detective – find which agent broke your multi-agent pipeline", "summary": "Agent Detective, a new open-source evaluation framework for multi-agent systems, identifies the first node where quality breaks in a pipeline by ingesting standard OpenTelemetry traces and rebuilding the execution graph. The tool, which is self-hosted and free, outputs a verdict with the origin, propagation path, and downstream cost, and can gate CI on an exit code without requiring a database, broker, or LLM by default.", "body_md": "Self-hosting stays free.\n\n[to be first to know.]Watch releases →\n\nClassifications still move between minor versions (\n\n[0.2.0 reclassified what 0.1.0 called]), so gate CI on the`loop_detected`\n\nexit code—`0`\n\nclean,`1`\n\nincident,`2`\n\ncould not run — never on an exact`report_type`\n\n.\n\n**An eval framework for multi-agent systems that names the culprit.** Ingest\nstandard OpenTelemetry traces, rebuild the execution graph, and find the\nfirst node where quality broke — the origin, the propagation path, and the\ndownstream cost. OTEL-native: any OpenInference / OpenLLMetry instrumented\nagent works with no code change.\n\n```\npip install agent-detective\ndetective analyze trace.json\n── graph 3f2a91c8  [content-pipeline]\n   FAILED  ·  cut_point  ·  confidence 62%\n\n   Origin — where quality broke\n     translator\n\n   Defects\n     ● Contract breach — translator\n       A carried input/output parameter was silently rewritten at translator.\n       observation 100% · attribution 92% · channel deterministic\n```\n\n| You want to… | Read |\n|---|---|\n| The guided tour — instrumenting, analyzing, CI, machine output |\n|\n| Connect your own agents (attributes, adapters, SDK) |\n|\n\n[docs/architecture.md](/Thomeras/agent_detective/blob/main/docs/architecture.md)[docs/capabilities.md](/Thomeras/agent_detective/blob/main/docs/capabilities.md),[docs/trace-requirements.md](/Thomeras/agent_detective/blob/main/docs/trace-requirements.md)[packages/detective_cli/README.md](/Thomeras/agent_detective/blob/main/packages/detective_cli/README.md)\n\n``` php\nflowchart LR\n    AG[\"your agent\"] -- \"OTLP/HTTP JSON\" --> CAP[\"detective capture :8900\"]\n    AG -- \"AGENT_DETECTIVE_TRACE_FILE\" --> F[\"run.json\"]\n    CAP --> AN[\"detective analyze\"]\n    F --> AN\n    AN --> V[\"verdict · exit 1 on incident<br/>terminal · --json · --markdown\"]\ndetective capture --once --out run.json # receive a trace straight from your agent\ndetective analyze run.json              # the verdict; exit 1 on an incident (CI gate as-is)\ndetective analyze run.json --markdown   # findings brief for a coding agent\ndetective doctor run.json               # is the trace even worth trusting?\n```\n\nAlready instrumented? Point the exporter at it:\n`OTEL_EXPORTER_OTLP_PROTOCOL=http/json OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:8900`\n\n.\nNot yet? `detective-sdk`\n\n(pure stdlib, zero dependencies) emits the same spans:\n\n``` python\nfrom detective_sdk import run\n\nwith run(\"intel\", task=user_request) as r:\n    with r.step(\"write\") as s:\n        s.output = dossier_markdown            # the work, not {\"ok\": true}\n        s.cost(usd=0.03, tokens_in=8_000, tokens_out=900, model=\"gpt-4o\")\n```\n\nNo database, no broker, and by default no LLM: the deterministic evidence\nchannel needs nothing but the trace. Set `JUDGE_BASE_URL`\n\n/ `JUDGE_MODEL`\n\n(any OpenAI-compatible endpoint — OpenAI, OpenRouter, local Ollama; concrete\nconfigs in [docs/usage.md](/Thomeras/agent_detective/blob/main/docs/usage.md#22-turning-on-the-per-node-judge)) to\nalso turn on the per-node quality judge — without it, nodes report *unscored*,\nnever silently \"fine\".\n\n[examples/diamond_eval.py](/Thomeras/agent_detective/blob/main/examples/diamond_eval.py) instruments a diamond —\none extractor, two parallel writers, one editor — and gates on the verdict.\nEach writer declares the facts it was handed as a **contract**; `--inject`\n\nmakes the marketing branch silently rewrite the price:\n\n```\nwith r.branch(\"marketing_writer\", input=facts) as s:\n    s.contract(price=facts[\"price\"], availability=facts[\"availability\"])\n    s.output = write_marketing(facts, inject)    # --inject rewrites the price\nbash\n$ python examples/diamond_eval.py --inject\ngraph 0a68bb06: cut_point — culprit: marketing_writer    # exit code 1\n```\n\nThe verdict names the branch that rewrote the fact — not the editor who merged it downstream — and verifies the breach actually shipped, all with no LLM:\n\n```\nFAILED  ·  cut_point  ·  confidence 95%\n\nOrigin — where quality broke\n  marketing_writer\n\n● Contract breach — marketing_writer\n  supporting: contract_breach (rule input_contract:price)\n  supporting: breach_propagated at terminal (contract-propagation check on the deliverable)\n\nDeterministic signals\n  fail contract_violation (marketing_writer) price: $12/user/month → from $5/user/month\n```\n\nPoint the same script at the full stack and the graph appears in the web UI —\nculprit ring on `marketing_writer`\n\n, propagation path into `press_editor`\n\n:\n\n```\nAGENT_DETECTIVE_ENDPOINT=http://localhost:8001 python examples/diamond_eval.py --inject\ndocker compose up --build                 # infra + ingest + worker + api + web\n./demo/run.sh                             # happy-path demo: graph, no incident\n./demo/inject_fault.sh && ./demo/run.sh   # a cut_point incident appears\n```\n\nWeb UI at `:5173`\n\n, read API at `:8000`\n\n, ingest at `:8001`\n\n; the bundled mock\nLLM judges, so no external API keys are needed.\n\nThe self-hosted stack has no authentication yet.Every host binding is therefore on`127.0.0.1`\n\nby default — the UI and API, and Postgres, ClickHouse, Redis and MinIO with them. What sits behind those ports is your traces' payloads: agent inputs and outputs verbatim, which is exactly the material you would not want public. Reaching the stack from elsewhere is one deliberate knob, and only belongs behind something that authenticates:\n\n```\nBIND=0.0.0.0 docker compose up --build\n```\n\nAuth and multi-tenancy are what Cloud is being built for.\n\n``` php\nflowchart LR\n    AG[\"your agents\"] -- \"OTLP/HTTP\" --> ING[\"ingest :8001\"]\n    ING --> CH[(\"ClickHouse<br/>raw spans\")]\n    ING --> PG[(\"Postgres<br/>graphs · runs · edges\")]\n    ING -.->|\"payload overflow\"| MIO[(\"MinIO\")]\n    ING -- \"Redis Streams\" --> T1[\"worker tier1<br/>cheap checks + terminal judge\"]\n    T1 -- \"flagged / sampled\" --> T2[\"worker tier2<br/>per-node scoring + judge\"]\n    T2 --> BE[\"blame engine<br/>(pure networkx)\"]\n    BE --> INC[(\"incidents +<br/>blame reports\")]\n    INC --> API[\"read API :8000\"]\n    API --> UI[\"web UI :5173\"]\n```\n\nThe report separates **where quality broke** (origin) from **where it\nsurfaced** (manifestation), flags rubber-stamping verifiers, and reports\ncapped, split confidence (observation × attribution). What the trace did not\ncapture, no analysis can manufacture — absent evidence renders `unverified`\n\n,\nnever `ok`\n\n.\n\nYour agent's process needs only ** detective-sdk** (25 KiB, zero\ndependencies): it emits the trace and never computes anything. The verdict\nhappens wherever\n\n**is installed — your laptop, CI, or the deployed stack — and that one install brings the whole analysis side with it (**\n\n`agent-detective`\n\n`otel-mapper`\n\nspan mapping, `agent-detective-worker`\n\npipeline,\n`blame-engine`\n\nverdicts, the CLI). `detective-ci`\n\nis a separate, opt-in\ninstall because it registers a pytest plugin. Details per distribution:\n[docs/usage.md §4.4](/Thomeras/agent_detective/blob/main/docs/usage.md#44-packages-and-distributions).\n\n```\npackages/\n  blame_engine/    pure, I/O-free blame analysis (networkx only)\n  otel_mapper/     OTLP span -> AgentRun/Edge mapping (Apache-2.0)\n  detective_cli/   the `agent-detective` pip distribution: local mode + CLI\n  detective_sdk/   zero-dependency instrumentation helpers\n  detective_ci/    deterministic golden replay + pytest plugin\nservices/          ingest · worker (tier1/tier2 + judge) · read API\ndb/                Alembic migrations        docker/clickhouse/  ClickHouse init\nweb/               React + Vite + cytoscape UI\nuv sync --all-packages --all-groups   # install the workspace\n./scripts/test.sh                     # every unit suite (mirrors CI)\nuv run pytest tests/e2e               # acceptance test (needs a running stack)\n\nfor p in blame-engine otel-mapper detective-sdk detective-ci \\\n         agent-detective-worker agent-detective; do\n  uv build --package \"$p\"             # the pip distributions\ndone\n```\n\nBusiness Source License 1.1 (see [LICENSE](/Thomeras/agent_detective/blob/main/LICENSE));\n`packages/otel_mapper`\n\nis Apache-2.0.", "url": "https://wpnews.pro/news/agent-detective-find-which-agent-broke-your-multi-agent-pipeline", "canonical_source": "https://github.com/Thomeras/agent_detective", "published_at": "2026-07-29 18:07:02+00:00", "updated_at": "2026-07-29 18:24:13.949905+00:00", "lang": "en", "topics": ["ai-agents", "developer-tools", "ai-tools"], "entities": ["Agent Detective", "OpenTelemetry", "OpenInference", "OpenLLMetry", "OpenAI", "OpenRouter", "Ollama"], "alternates": {"html": "https://wpnews.pro/news/agent-detective-find-which-agent-broke-your-multi-agent-pipeline", "markdown": "https://wpnews.pro/news/agent-detective-find-which-agent-broke-your-multi-agent-pipeline.md", "text": "https://wpnews.pro/news/agent-detective-find-which-agent-broke-your-multi-agent-pipeline.txt", "jsonld": "https://wpnews.pro/news/agent-detective-find-which-agent-broke-your-multi-agent-pipeline.jsonld"}}