Signed OpenTelemetry GenAI spans for AI agents. Capture, normalize, verify — bring your own backend.
Trail is a Python SDK that captures what AI agents actually do — every LLM call, tool invocation, MCP call, and skill execution — as OpenTelemetry spans with a small Trail extension namespace. It signs each session with Ed25519 and exports via OTLP to any OTel backend (Grafana, Honeycomb, Datadog, Chronosphere, ...). Trail does not store, query, or dashboard. Storage and query are your existing backend's job.
When an agent misbehaves in production, three questions are surprisingly hard to answer:
Which tool ran, in what order, with what inputs? Existing tracers are LLM-call-shaped, not agent-shaped.Was that MCP server response trying to inject instructions? No mainstream tracer flags this.Is this skill the same code it was yesterday? Skill substitution leaves no trace by default.
Trail adds the three things that are missing: an agent-aware tool taxonomy (internal
/ mcp
/ skill
/ builtin
), MCP injection flagging on tool responses, and a skill hash that detects silent substitution — all as standard OpenTelemetry spans, so any OTel backend ingests them with no translation layer.
Trail models an agent run as an OpenTelemetry span tree — one invoke_agent
root span per session, with every LLM call, tool, MCP call, and skill nested underneath — and layers a trail.*
attribute namespace on top. That structure, plus three purpose-built attributes, is what turns each question above into a query.
Which tool ran, in what order, with what inputs?
Every tool invocation becomes an execute_tool
span tagged with gen_ai.tool.name
and trail.tool_type
(internal
/ mcp
/ skill
/ builtin
) — the agent-shaped distinction a plain LLM tracer never draws. Order and nesting come from the OpenTelemetry SDK's contextvars
propagation, which stays correct across async
/await
and concurrent asyncio
tasks, so each span attaches to the right parent. Inputs and outputs are recorded as trail.input_hash
/ trail.output_hash
(SHA-256, computed off the hot path) plus a sensitivity flag — tamper-evident identity of the payloads without storing the payloads themselves.
gen_ai.operation.name = "execute_tool"
gen_ai.tool.name = "get_customer_record"
trail.tool_type = "mcp"
trail.input_hash = "sha256:..."
Was that MCP server response trying to inject instructions?
When Trail wraps an MCP call_tool
, it runs the response through a YAML injection ruleset — instruction-override, system-prompt injection, role override, credential-exfil phrasing (override via TRAIL_MCP_RULES
) — and stamps the span with trail.mcp.injection_flag
. A response that says "ignore your previous instructions and…" lands as an ordinary span with trail.mcp.injection_flag = true
, next to trail.mcp.server_id
for provenance.
Is this skill the same code it was yesterday?
wrap_skill()
records trail.skill.hash
— a SHA-256 over the skill's source (trail.skill.hash_method = "source"
, with a qualname-fallback
for C-extensions and lambdas). Same skill → same hash; a silent swap → a different hash on today's span versus yesterday's. Diff the attribute across two sessions and substitution is visible.
Then you ask where you already look. Trail only captures — the questions get answered in your backend. In dev mode that's the session JSONL (~/.trail/sessions/{trace_id}.jsonl
), and trail verify-export
proves none of it was altered after the fact (and pinpoints the span if it was). In prod, it's an ordinary attribute filter — trail.tool_type = "mcp" AND trail.mcp.injection_flag = true
— in Grafana, Honeycomb, or Datadog.
pip install 'trail-otel[openai]'
python
import openai
import trail
trail.auto_instrument() # detects openai, instruments it
with trail.session(agent_id="content-pipeline"):
client = openai.OpenAI()
client.chat.completions.create(model="gpt-4o", messages=[...])
That's it. By default Trail writes spans to ~/.trail/sessions/{trace_id}.jsonl
and a short summary to stderr. Zero infrastructure.
To ship to your OTel backend instead:
export TRAIL_EXPORT=otlp
export OTEL_EXPORTER_OTLP_ENDPOINT=https://your-collector:4317
Async OpenAI (AsyncOpenAI
) is instrumented automatically by the same trail.auto_instrument()
call.
Trail ships a trail-hook
console script. Wire it into ~/.claude/settings.json
. A single binary handles all three events — it reads the event name from Claude Code's stdin payload and dispatches internally.
{
"hooks": {
"PreToolUse": [
{ "matcher": "*", "hooks": [{ "type": "command", "command": "trail-hook" }] }
],
"PostToolUse": [
{ "matcher": "*", "hooks": [{ "type": "command", "command": "trail-hook" }] }
],
"SessionEnd": [
{ "hooks": [{ "type": "command", "command": "trail-hook" }] }
]
}
}
Already have hooks? Claude Code's hooks.<EventName>
is an array — Trail composes alongside whatever is already there. Append a new {matcher, hooks}
block per event rather than replacing the array. Trail runs sequentially with your existing hooks and never blocks them (it exits 0 even on internal errors).
See it end to end: examples/demo_claude_code/run_demo.sh
replays a real Claude Code session against trail-hook
(no infrastructure, no API key) — captures the tool taxonomy, flags a prompt-injection riding in on an MCP-fetched GitHub issue, then verify-export
proves the session and catches a tamper.
Every Claude Code tool call — including MCP calls and skills — is now captured. The SessionEnd
hook is the moment the session gets its Merkle root + Ed25519 signature. (Signing is wired to SessionEnd
, which fires once when the session terminates — not Stop
, which fires at the end of every turn and would leave later turns' spans unsigned.)
Google's Agent Development Kit is OpenTelemetry-native, so Trail rides ADK's own
execute_tool
spans rather than re-instrumenting — one auto_instrument()
call
adds the tool taxonomy and MCP injection flag ADK doesn't produce, and wrapping
the run in trail.session()
signs it.
import trail
from google.adk.runners import Runner
trail.auto_instrument() # detects google.adk, enriches its tool spans
with trail.session(agent_id="support-triage", provider="gcp.vertex"):
runner.run(user_id="u1", session_id="s1", new_message=msg)
Every ADK tool call now carries trail.tool_type
(McpTool
→ mcp
,
ADK-provided search/memory tools → builtin
, your FunctionTool
s → internal
)
and MCP responses are scanned for injection (trail.mcp.injection_flag
).
Try it in dev mode first — zero infrastructure. Dev mode is the default, so
the two lines above already write every ADK span to
~/.trail/sessions/{trace_id}.jsonl
locally (no network). Run your agent, then inspect what was captured:
cat ~/.trail/sessions/<trace_id>.jsonl | jq . # spans + trail.tool_type
Signing is opt-in. With no keys present, sessions are simply unsigned —
the minimal setup: spans + trail.tool_type
- MCP injection flag, no tamper-evidence, no signing overhead. Turn it on when you want it:
trail generate-keys # once; enables signing
trail verify-export ~/.trail/sessions/<trace_id>.jsonl
Dev-mode note:don't also enable ADK's own Cloud Trace / OTel exporter while running dev mode. Trail configures the tracer provider; if ADK sets one first, Trail's local JSONL won't attach. Just add the two Trail lines and leave ADK's own tracing off.
When it looks right locally, ship the same code to your backend — ADK exports
OTLP, so set TRAIL_EXPORT=otlp
and OTEL_EXPORTER_OTLP_ENDPOINT
(e.g. Chronosphere) and the spans flow there instead. See docs/backends/chronosphere.md, and
for the framework-agnostic manual path (no adapter required).
examples/adk_manual_instrumentation.py
ADK has no first-class "skill", so skill-hashing stays with
trail.wrap_skill
, which composes with ADK. Parallel/merged tool calls are a documented v1 gap.
Standard OpenTelemetry GenAI attributes:
gen_ai.operation.name = "chat" | "execute_tool" | "invoke_agent"
gen_ai.provider.name = "openai" | "anthropic"
gen_ai.request.model = "gpt-4o"
gen_ai.tool.name = "get_customer_record"
gen_ai.usage.input_tokens = 1240
Plus the Trail extension — the novel part:
trail.tool_type = "internal" | "mcp" | "skill" | "builtin"
trail.mcp.server_id = "acme-crm-mcp"
trail.mcp.injection_flag = false
trail.skill.hash = "sha256:..."
trail.input_hash = "sha256:..."
trail.output_hash = "sha256:..."
In Grafana or Honeycomb, these render as ordinary GenAI spans. The trail.*
attributes are queryable like any other attribute (trail.tool_type = "mcp" AND trail.mcp.injection_flag = true
).
Trail emits spans only — no Prometheus scrape endpoint and no OTel metrics. To get rate / error / duration counters or a "MCP injections per minute" panel, drop the OpenTelemetry Collector's spanmetrics
connector into your pipeline and label by trail.tool_type
, trail.mcp.injection_flag
, etc. Span backends (Tempo's metrics-generator, Datadog APM metrics, Honeycomb derived columns) offer equivalent backend-side derivations. Two signal types at the source would duplicate the signal — the Collector composes them cleanly.
Each session is signed once at session end with Ed25519 over a Merkle root of its span content. Anyone with the public key can verify it later — no Trail infrastructure required:
trail verify-export session.jsonl
Modified spans, removed spans, and added spans are all detected by the Merkle root mismatch.
Generate a keypair:
trail generate-keys
| Mode | Storage | Signing | Network | Use it for |
|---|---|---|---|---|
| Dev (default) | ~/.trail/sessions/*.jsonl + stderr summary |
|||
| Off | None | Local debugging | ||
| Prod | OTLP to your backend | On (Ed25519 + Merkle, at session end) | OTLP | Shipping to Grafana / Honeycomb / Datadog / Chronosphere |
Per-backend setup (endpoint, auth, query examples) lives in docs/backends/ — Grafana Tempo, Honeycomb, Datadog, Chronosphere. For clusters, see
docs/deployment/kubernetes.md
In: OpenAI SDK adapter, Google ADK adapter, Claude Code hooks, OTel GenAI emission, tool taxonomy, MCP injection flagging, skill hash, session-checkpoint signing, OTLP transport, dev-mode JSONL, verify-export
, generate-keys
.
Not yet: LangChain / LlamaIndex / AutoGen adapters, HTTP proxy, sidecar deployment, CloudTrail / CloudWatch transports, encrypted sensitive-content side-store, GDPR erasure workflow, multi-org config, KMS-backed signing.
Known v1 limitations:
- In-process capture is suppressible by the agent code. Trail v1 is positioned as a developer debugging tool. Suppression-resistant capture (proxy / sidecar) is a v2 theme. - A process crash before session end leaves spans unsigned(still exported, just unverifiable). Per-event signing is v2. - Claude Code hooks expose tool events, not LLM calls — so the LLM-token detail you'd get from the OpenAI adapter is absent from the Claude Code path. Tool taxonomy, MCP flagging, and skill hash come through on both paths.
Suppression-resistant capture (HTTP proxy + sidecar), per-event / checkpoint signing for crash safety, additional framework adapters (LangChain, LlamaIndex, AutoGen), encrypted sensitive-content side-store, GDPR erasure workflow, KMS-backed signing, additional transports (CloudTrail, CloudWatch).
See trail_hld.md
for the high-level design and CLAUDE.md
for implementation conventions.
Trail produces signed, tamper-evident telemetry — reports against the signing / verification path are taken seriously. See SECURITY.md for the disclosure process and what is in scope.
Apache-2.0. See LICENSE
and NOTICE
.