Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability

A developer designed a run-centric trace model for LLM and agentic AI observability built on OpenTelemetry's GenAI semantic conventions (gen_ai.*), using one trace per run with three span types: invoke_agent, chat, and execute_tool. The design attaches token usage to every model-call span and error status to every failed tool call, which the author argues is enough to answer which path an agent took, what it cost, and what changed during an incident. A verified example agent with stubbed model and tool clients runs without an API key, and the writeup covers production trade-offs including sampling, cardinality, content capture, and context propagation across agents.

title: "Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability" published: false description: "A run-centric trace model for LLM and agentic AI observability with OpenTelemetry GenAI conventions, with measurements from a verified example." tags: opentelemetry, observability, devops, bonree LLM applications and agentic AI operations share a property that classic APM was not designed for: the execution path is decided at runtime by a model. A request that returns HTTP 200 may have looped, retried a failing tool, or consumed several times the usual number of tokens, and none of that is visible from the response. This article argues for a run-centric trace model, implements it with the OpenTelemetry GenAI semantic conventions gen ai. , and reports what the trace contains when an example agent is executed. It then covers the production trade-offs that decide whether such tracing survives real traffic: sampling, cardinality, content capture and context propagation across agents. Conclusion. One trace per run, three span types invoke agent , chat , execute tool , token usage on every model-call span and error status on every failed tool call are sufficient to answer three of the four questions that recur in agent incidents: which path was taken, what it cost, and what was changed. The fourth, whether the answer was correct, needs an evaluation signal that tracing cannot supply. Four properties distinguish an agent run from a request: The consequence is that the entity to observe is the run, and its internal structure matters more than its endpoint. | Span name | gen ai.operation.name | Represents | |---|---|---| | invoke agent {agent} | invoke agent | The whole run | | chat {model} | chat | One model request | | execute tool {tool} | execute tool | One tool invocation | Three design choices are worth stating explicitly. app. , so that a later addition to the specification cannot collide with it. The listing is complete and runs without an API key. The model client and the tools are stubs, and the refund tool is configured to fail once and then succeed, so every run is identical. python import time from opentelemetry import trace from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter provider = TracerProvider resource=Resource.create {"service.name": "support-agent"} provider.add span processor BatchSpanProcessor ConsoleSpanExporter trace.set tracer provider provider tracer = trace.get tracer "support-agent" PROVIDER = "example-provider" MODEL = "example-model" def call llm prompt: str - dict: """Stand-in for a real model client.""" time.sleep 0.2 return { "text": "...", "input tokens": len prompt.split 2, "output tokens": 48, "finish reason": "stop", } def traced chat prompt: str - dict: with tracer.start as current span f"chat {MODEL}" as span: span.set attribute "gen ai.operation.name", "chat" span.set attribute "gen ai.provider.name", PROVIDER span.set attribute "gen ai.request.model", MODEL resp = call llm prompt span.set attribute "gen ai.usage.input tokens", resp "input tokens" span.set attribute "gen ai.usage.output tokens", resp "output tokens" span.set attribute "gen ai.response.finish reasons", resp "finish reason" return resp def traced tool name: str, fn, args : An exception raised inside the span is recorded and the span status is set to ERROR automatically by the SDK; we only add the semantic-convention error.type attribute so failures can be grouped by class. with tracer.start as current span f"execute tool {name}" as span: span.set attribute "gen ai.operation.name", "execute tool" span.set attribute "gen ai.tool.name", name try: return fn args except Exception as exc: span.set attribute "error.type", type exc . name raise def lookup order order id: str - dict: return {"order id": order id, "status": "delivered"} transient failures = {"issue refund": 1} simulate one timeout, then success def issue refund order id: str - dict: if transient failures "issue refund" 0: transient failures "issue refund" -= 1 raise TimeoutError "upstream refund API timed out" return {"refunded": True} def run agent question: str - None: with tracer.start as current span "invoke agent support-agent" as root: root.set attribute "gen ai.operation.name", "invoke agent" root.set attribute "gen ai.provider.name", PROVIDER root.set attribute "gen ai.agent.name", "support-agent" model calls = 0 traced chat question model calls += 1 traced tool "lookup order", lookup order, order id="A-1042" traced chat "order found, decide next step" model calls += 1 for attempt in range 1, 3 : try: traced tool "issue refund", issue refund, order id="A-1042" break except TimeoutError: root.add event "tool retry", {"tool": "issue refund", "attempt": attempt} traced chat "write the final answer" model calls += 1 root.set attribute "app.agent.model calls", model calls custom attribute if name == " main ": run agent "Where is my order A-1042, and can I get a refund?" Four notes on the listing: ERROR . The only addition is error.type , which makes failures groupable by class. Running the listing produces seven spans: one invoke agent , three chat and three execute tool . | Span | Status | Notes | |---|---|---| | chat example-model 1st | unset | 22 input / 48 output tokens | | execute tool lookup order | unset | | | chat example-model 2nd | unset | 10 input / 48 output tokens | | execute tool issue refund 1st | ERROR | error.type = TimeoutError , exception event recorded | | execute tool issue refund 2nd | unset | The retry succeeded | | chat example-model 3rd | unset | 8 input / 48 output tokens | | invoke agent support-agent | unset | tool retry event; app.agent.model calls = 3 | Three observations follow directly from this data. chat spans. In this example the model stub dominates run time by construction. In a real system, the split between model time and tool time is exactly what the trace is meant to reveal. Sampling. Head-based sampling decides before a run starts and therefore discards the rare runs that matter: failures, loops and unusually expensive runs. Tail-based sampling for example, the OpenTelemetry Collector's tail sampling processor decides after the run completes. It can keep every run with an error status, a high model-call count or high token usage, plus a small fraction of ordinary runs as a baseline. Cardinality. Identifiers such as conversation IDs or order numbers belong on spans, not on metric dimensions. Derive metrics from spans with bounded labels: agent name, model, tool name and error.type . The GenAI conventions also define client metrics for token usage and operation duration, which fit this pattern. Content capture. Prompts and completions are large and often sensitive. Keep capture off by default, enable it per environment behind redaction, and prefer storing a reference to the payload over the payload itself. Context propagation across agents. When one agent hands work to another over HTTP or a message queue, propagate the W3C trace context in the request headers or message metadata. Otherwise the run fragments into disconnected traces, and the seam, where multi-agent failures tend to occur, becomes the part that cannot be seen. Specification drift. Parts of the GenAI conventions are still experimental. Keep attribute names in one module so that a rename is a one-line change, and pin the instrumentation version. Agents that operate production systems, for example to perform root cause analysis, are both consumers and subjects of observability data. As consumers, their conclusions should reference the telemetry they rely on topology, metrics, errors , so that an engineer can verify a diagnosis instead of repeating it. As subjects, their own runs need the trace structure above, plus a record of which actions were proposed, approved and executed. Determinism cannot be demanded of the model, but it can be demanded of the system around it: pinned model and prompt versions, typed tools with narrow permissions, and replayable runs. Bonree ONE's AI Observability applies this model to applications built on LangChain, LangGraph, Dify and OpenClaw, and places the agent data next to the services beneath it. Bonree ONE Sage AI, Bonree's agent operations platform, builds its fault diagnosis on topology, metric and error data of the kind described here. invoke agent with execute tool children. A broader checklist for reviewing agent instrumentation is available at agent-trace-checklist https://github.com/Bonree-official/Technology-Blog/blob/main/Agentic-AI-Operations-Evidence-Carrying-Handoffs.md . Disclosure: I am part of the Bonree team. The code above works with any OTLP-compatible backend.