{"slug": "trail-signed-opentelemetry-spans-for-ai-agents", "title": "Trail – signed OpenTelemetry spans for AI agents", "summary": "Trail, a new Python SDK, captures AI agent activity as signed OpenTelemetry spans, adding agent-aware tool taxonomy, MCP injection flagging, and skill hashing to detect silent substitution, while exporting via OTLP to any backend without storing data itself. The SDK signs each session with Ed25519 and supports backends like Grafana, Honeycomb, Datadog, and Chronosphere, aiming to answer questions about tool execution order, prompt injection attempts, and skill code changes in production.", "body_md": "**Signed OpenTelemetry GenAI spans for AI agents. Capture, normalize, verify — bring your own backend.**\n\nTrail is a Python SDK that captures what AI agents actually do — every LLM call, tool invocation, MCP call, and skill execution — as OpenTelemetry spans with a small Trail extension namespace. It signs each session with Ed25519 and exports via OTLP to any OTel backend (Grafana, Honeycomb, Datadog, Chronosphere, ...). Trail does not store, query, or dashboard. Storage and query are your existing backend's job.\n\nWhen an agent misbehaves in production, three questions are surprisingly hard to answer:\n\n**Which tool ran, in what order, with what inputs?** Existing tracers are LLM-call-shaped, not agent-shaped.**Was that MCP server response trying to inject instructions?** No mainstream tracer flags this.**Is this skill the same code it was yesterday?** Skill substitution leaves no trace by default.\n\nTrail adds the three things that are missing: an **agent-aware tool taxonomy** (`internal`\n\n/ `mcp`\n\n/ `skill`\n\n/ `builtin`\n\n), **MCP injection flagging** on tool responses, and a **skill hash** that detects silent substitution — all as standard OpenTelemetry spans, so any OTel backend ingests them with no translation layer.\n\nTrail models an agent run as an OpenTelemetry **span tree** — one `invoke_agent`\n\nroot span per session, with every LLM call, tool, MCP call, and skill nested underneath — and layers a `trail.*`\n\nattribute namespace on top. That structure, plus three purpose-built attributes, is what turns each question above into a query.\n\n**Which tool ran, in what order, with what inputs?**\nEvery tool invocation becomes an `execute_tool`\n\nspan tagged with `gen_ai.tool.name`\n\nand `trail.tool_type`\n\n(`internal`\n\n/ `mcp`\n\n/ `skill`\n\n/ `builtin`\n\n) — the agent-shaped distinction a plain LLM tracer never draws. Order and nesting come from the OpenTelemetry SDK's `contextvars`\n\npropagation, which stays correct across `async`\n\n/`await`\n\nand concurrent `asyncio`\n\ntasks, so each span attaches to the right parent. Inputs and outputs are recorded as `trail.input_hash`\n\n/ `trail.output_hash`\n\n(SHA-256, computed off the hot path) plus a sensitivity flag — tamper-evident identity of the payloads without storing the payloads themselves.\n\n```\ngen_ai.operation.name = \"execute_tool\"\ngen_ai.tool.name      = \"get_customer_record\"\ntrail.tool_type       = \"mcp\"\ntrail.input_hash      = \"sha256:...\"\n```\n\n**Was that MCP server response trying to inject instructions?**\nWhen Trail wraps an MCP `call_tool`\n\n, it runs the *response* through a YAML injection ruleset — instruction-override, system-prompt injection, role override, credential-exfil phrasing (override via `TRAIL_MCP_RULES`\n\n) — and stamps the span with `trail.mcp.injection_flag`\n\n. A response that says \"ignore your previous instructions and…\" lands as an ordinary span with `trail.mcp.injection_flag = true`\n\n, next to `trail.mcp.server_id`\n\nfor provenance.\n\n**Is this skill the same code it was yesterday?**\n`wrap_skill()`\n\nrecords `trail.skill.hash`\n\n— a SHA-256 over the skill's source (`trail.skill.hash_method = \"source\"`\n\n, with a `qualname-fallback`\n\nfor C-extensions and lambdas). Same skill → same hash; a silent swap → a different hash on today's span versus yesterday's. Diff the attribute across two sessions and substitution is visible.\n\n**Then you ask where you already look.** Trail only captures — the questions get *answered* in your backend. In dev mode that's the session JSONL (`~/.trail/sessions/{trace_id}.jsonl`\n\n), and `trail verify-export`\n\nproves none of it was altered after the fact (and pinpoints the span if it was). In prod, it's an ordinary attribute filter — `trail.tool_type = \"mcp\" AND trail.mcp.injection_flag = true`\n\n— in Grafana, Honeycomb, or Datadog.\n\n```\npip install 'trail-otel[openai]'\npython\nimport openai\nimport trail\n\ntrail.auto_instrument()           # detects openai, instruments it\n\nwith trail.session(agent_id=\"content-pipeline\"):\n    client = openai.OpenAI()\n    client.chat.completions.create(model=\"gpt-4o\", messages=[...])\n```\n\nThat's it. By default Trail writes spans to `~/.trail/sessions/{trace_id}.jsonl`\n\nand a short summary to stderr. Zero infrastructure.\n\nTo ship to your OTel backend instead:\n\n```\nexport TRAIL_EXPORT=otlp\nexport OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector:4317\n```\n\nAsync OpenAI (`AsyncOpenAI`\n\n) is instrumented automatically by the same `trail.auto_instrument()`\n\ncall.\n\nTrail ships a `trail-hook`\n\nconsole script. Wire it into `~/.claude/settings.json`\n\n. A single binary handles all three events — it reads the event name from Claude Code's stdin payload and dispatches internally.\n\n```\n{\n  \"hooks\": {\n    \"PreToolUse\": [\n      { \"matcher\": \"*\", \"hooks\": [{ \"type\": \"command\", \"command\": \"trail-hook\" }] }\n    ],\n    \"PostToolUse\": [\n      { \"matcher\": \"*\", \"hooks\": [{ \"type\": \"command\", \"command\": \"trail-hook\" }] }\n    ],\n    \"SessionEnd\": [\n      { \"hooks\": [{ \"type\": \"command\", \"command\": \"trail-hook\" }] }\n    ]\n  }\n}\n```\n\n**Already have hooks?** Claude Code's `hooks.<EventName>`\n\nis an array — Trail composes alongside whatever is already there. Append a new `{matcher, hooks}`\n\nblock per event rather than replacing the array. Trail runs sequentially with your existing hooks and never blocks them (it exits 0 even on internal errors).\n\n**See it end to end:** `examples/demo_claude_code/run_demo.sh`\n\nreplays a real Claude Code session against `trail-hook`\n\n(no infrastructure, no API key) — captures the tool taxonomy, flags a prompt-injection riding in on an MCP-fetched GitHub issue, then `verify-export`\n\nproves the session and catches a tamper.\n\nEvery Claude Code tool call — including MCP calls and skills — is now captured. The `SessionEnd`\n\nhook is the moment the session gets its Merkle root + Ed25519 signature. (Signing is wired to `SessionEnd`\n\n, which fires once when the session terminates — **not** `Stop`\n\n, which fires at the end of every turn and would leave later turns' spans unsigned.)\n\nGoogle's Agent Development Kit is OpenTelemetry-native, so Trail rides ADK's own\n`execute_tool`\n\nspans rather than re-instrumenting — one `auto_instrument()`\n\ncall\nadds the tool taxonomy and MCP injection flag ADK doesn't produce, and wrapping\nthe run in `trail.session()`\n\nsigns it.\n\n``` python\nimport trail\nfrom google.adk.runners import Runner\n\ntrail.auto_instrument()           # detects google.adk, enriches its tool spans\n\nwith trail.session(agent_id=\"support-triage\", provider=\"gcp.vertex\"):\n    runner.run(user_id=\"u1\", session_id=\"s1\", new_message=msg)\n```\n\nEvery ADK tool call now carries `trail.tool_type`\n\n(`McpTool`\n\n→ `mcp`\n\n,\nADK-provided search/memory tools → `builtin`\n\n, your `FunctionTool`\n\ns → `internal`\n\n)\nand MCP responses are scanned for injection (`trail.mcp.injection_flag`\n\n).\n\n**Try it in dev mode first — zero infrastructure.** Dev mode is the default, so\nthe two lines above already write every ADK span to\n`~/.trail/sessions/{trace_id}.jsonl`\n\nlocally (no network). Run your agent, then\ninspect what was captured:\n\n```\ncat ~/.trail/sessions/<trace_id>.jsonl | jq .      # spans + trail.tool_type\n```\n\n**Signing is opt-in.** With no keys present, sessions are simply *unsigned* —\nthe minimal setup: spans + `trail.tool_type`\n\n+ MCP injection flag, no\ntamper-evidence, no signing overhead. Turn it on when you want it:\n\n```\ntrail generate-keys                                # once; enables signing\ntrail verify-export ~/.trail/sessions/<trace_id>.jsonl\n# → VALID  (N spans, signature valid, key fpr ...)\n```\n\nDev-mode note:don't also enable ADK's own Cloud Trace / OTel exporter while running dev mode. Trail configures the tracer provider; if ADK sets one first, Trail's local JSONL won't attach. Just add the two Trail lines and leave ADK's own tracing off.\n\nWhen it looks right locally, ship the *same* code to your backend — ADK exports\nOTLP, so set `TRAIL_EXPORT=otlp`\n\nand `OTEL_EXPORTER_OTLP_ENDPOINT`\n\n(e.g.\nChronosphere) and the spans flow there instead. See\n[ docs/backends/chronosphere.md](/varmax2511/trail/blob/main/docs/backends/chronosphere.md), and\n\n[for the framework-agnostic manual path (no adapter required).](/varmax2511/trail/blob/main/examples/adk_manual_instrumentation.py)\n\n`examples/adk_manual_instrumentation.py`\n\nADK has no first-class \"skill\", so skill-hashing stays with\n\n`trail.wrap_skill`\n\n, which composes with ADK. Parallel/merged tool calls are a documented v1 gap.\n\nStandard OpenTelemetry GenAI attributes:\n\n```\ngen_ai.operation.name      = \"chat\" | \"execute_tool\" | \"invoke_agent\"\ngen_ai.provider.name       = \"openai\" | \"anthropic\"\ngen_ai.request.model       = \"gpt-4o\"\ngen_ai.tool.name           = \"get_customer_record\"\ngen_ai.usage.input_tokens  = 1240\n```\n\nPlus the Trail extension — the novel part:\n\n```\ntrail.tool_type            = \"internal\" | \"mcp\" | \"skill\" | \"builtin\"\ntrail.mcp.server_id        = \"acme-crm-mcp\"\ntrail.mcp.injection_flag   = false\ntrail.skill.hash           = \"sha256:...\"\ntrail.input_hash           = \"sha256:...\"\ntrail.output_hash          = \"sha256:...\"\n```\n\nIn Grafana or Honeycomb, these render as ordinary GenAI spans. The `trail.*`\n\nattributes are queryable like any other attribute (`trail.tool_type = \"mcp\" AND trail.mcp.injection_flag = true`\n\n).\n\nTrail emits **spans only** — no Prometheus scrape endpoint and no OTel metrics. To get rate / error / duration counters or a \"MCP injections per minute\" panel, drop the OpenTelemetry Collector's `spanmetrics`\n\nconnector into your pipeline and label by `trail.tool_type`\n\n, `trail.mcp.injection_flag`\n\n, etc. Span backends (Tempo's metrics-generator, Datadog APM metrics, Honeycomb derived columns) offer equivalent backend-side derivations. Two signal types at the source would duplicate the signal — the Collector composes them cleanly.\n\nEach session is signed once at session end with Ed25519 over a Merkle root of its span content. Anyone with the public key can verify it later — no Trail infrastructure required:\n\n```\ntrail verify-export session.jsonl\n# → VALID  (132 spans, signed 2026-06-06T10:02:14Z, key fpr sha256:abcd...)\n```\n\nModified spans, removed spans, and added spans are all detected by the Merkle root mismatch.\n\nGenerate a keypair:\n\n```\ntrail generate-keys\n# → ~/.trail/keys/trail.key  (private, chmod 600)\n# → ~/.trail/keys/trail.pub  (public)\n```\n\n| Mode | Storage | Signing | Network | Use it for |\n|---|---|---|---|---|\n| Dev (default) | `~/.trail/sessions/*.jsonl` + stderr summary |\nOff | None | Local debugging |\n| Prod | OTLP to your backend | On (Ed25519 + Merkle, at session end) | OTLP | Shipping to Grafana / Honeycomb / Datadog / Chronosphere |\n\nPer-backend setup (endpoint, auth, query examples) lives in\n[ docs/backends/](/varmax2511/trail/blob/main/docs/backends/README.md) — Grafana Tempo, Honeycomb, Datadog,\nChronosphere. For clusters, see\n\n[.](/varmax2511/trail/blob/main/docs/deployment/kubernetes.md)\n\n`docs/deployment/kubernetes.md`\n\n**In:** OpenAI SDK adapter, Google ADK adapter, Claude Code hooks, OTel GenAI emission, tool taxonomy, MCP injection flagging, skill hash, session-checkpoint signing, OTLP transport, dev-mode JSONL, `verify-export`\n\n, `generate-keys`\n\n.\n\n**Not yet:** LangChain / LlamaIndex / AutoGen adapters, HTTP proxy, sidecar deployment, CloudTrail / CloudWatch transports, encrypted sensitive-content side-store, GDPR erasure workflow, multi-org config, KMS-backed signing.\n\n**Known v1 limitations:**\n\n- In-process capture is\n**suppressible** by the agent code. Trail v1 is positioned as a developer debugging tool. Suppression-resistant capture (proxy / sidecar) is a v2 theme. - A process crash before session end leaves spans\n**unsigned**(still exported, just unverifiable). Per-event signing is v2. - Claude Code hooks expose tool events, not LLM calls — so the LLM-token detail you'd get from the OpenAI adapter is absent from the Claude Code path. Tool taxonomy, MCP flagging, and skill hash come through on both paths.\n\nSuppression-resistant capture (HTTP proxy + sidecar), per-event / checkpoint signing for crash safety, additional framework adapters (LangChain, LlamaIndex, AutoGen), encrypted sensitive-content side-store, GDPR erasure workflow, KMS-backed signing, additional transports (CloudTrail, CloudWatch).\n\nSee `trail_hld.md`\n\nfor the high-level design and `CLAUDE.md`\n\nfor implementation conventions.\n\nTrail produces signed, tamper-evident telemetry — reports against the signing /\nverification path are taken seriously. See [ SECURITY.md](/varmax2511/trail/blob/main/SECURITY.md) for the\ndisclosure process and what is in scope.\n\nApache-2.0. See `LICENSE`\n\nand `NOTICE`\n\n.", "url": "https://wpnews.pro/news/trail-signed-opentelemetry-spans-for-ai-agents", "canonical_source": "https://github.com/varmax2511/trail", "published_at": "2026-08-03 01:30:58+00:00", "updated_at": "2026-08-03 01:52:47.271740+00:00", "lang": "en", "topics": ["ai-agents", "ai-tools", "ai-safety", "developer-tools"], "entities": ["Trail", "OpenTelemetry", "Ed25519", "Grafana", "Honeycomb", "Datadog", "Chronosphere", "MCP"], "alternates": {"html": "https://wpnews.pro/news/trail-signed-opentelemetry-spans-for-ai-agents", "markdown": "https://wpnews.pro/news/trail-signed-opentelemetry-spans-for-ai-agents.md", "text": "https://wpnews.pro/news/trail-signed-opentelemetry-spans-for-ai-agents.txt", "jsonld": "https://wpnews.pro/news/trail-signed-opentelemetry-spans-for-ai-agents.jsonld"}}