Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability

title: "Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability"
published: false
description: "A run-centric trace model for LLM and agentic AI observability with OpenTelemetry GenAI conventions, with measurements from a verified example."
tags: opentelemetry, observability, devops, bonree

LLM applications and agentic AI operations share a property that classic APM was not designed for: the execution path is decided at runtime by a model. A request that returns HTTP 200 may have looped, retried a failing tool, or consumed several times the usual number of tokens, and none of that is visible from the response.

This article argues for a run-centric trace model, implements it with the OpenTelemetry GenAI semantic conventions (gen_ai.*), and reports what the trace contains when an example agent is executed. It then covers the production trade-offs that decide whether such tracing survives real traffic: sampling, cardinality, content capture and context propagation across agents.

Conclusion. One trace per run, three span types (invoke_agent, chat, execute_tool), token usage on every model-call span and error status on every failed tool call are sufficient to answer three of the four questions that recur in agent incidents: which path was taken, what it cost, and what was changed. The fourth, whether the answer was correct, needs an evaluation signal that tracing cannot supply.

Four properties distinguish an agent run from a request:

The consequence is that the entity to observe is the run, and its internal structure matters more than its endpoint.

Three design choices are worth stating explicitly.

app.*), so that a later addition to the specification cannot collide with it. The listing is complete and runs without an API key. The model client and the tools are stubs, and the refund tool is configured to fail once and then succeed, so every run is identical.

import time

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter

provider = TracerProvider(resource=Resource.create({"service.name": "support-agent"}))
provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("support-agent")

PROVIDER = "example-provider"
MODEL = "example-model"

def call_llm(prompt: str) -> dict:
    """Stand-in for a real model client."""
    time.sleep(0.2)
    return {
        "text": "...",
        "input_tokens": len(prompt.split()) * 2,
        "output_tokens": 48,
        "finish_reason": "stop",
    }

def traced_chat(prompt: str) -> dict:
    with tracer.start_as_current_span(f"chat {MODEL}") as span:
        span.set_attribute("gen_ai.operation.name", "chat")
        span.set_attribute("gen_ai.provider.name", PROVIDER)
        span.set_attribute("gen_ai.request.model", MODEL)
        resp = call_llm(prompt)
        span.set_attribute("gen_ai.usage.input_tokens", resp["input_tokens"])
        span.set_attribute("gen_ai.usage.output_tokens", resp["output_tokens"])
        span.set_attribute("gen_ai.response.finish_reasons", [resp["finish_reason"]])
        return resp

def traced_tool(name: str, fn, **args):
    with tracer.start_as_current_span(f"execute_tool {name}") as span:
        span.set_attribute("gen_ai.operation.name", "execute_tool")
        span.set_attribute("gen_ai.tool.name", name)
        try:
            return fn(**args)
        except Exception as exc:
            span.set_attribute("error.type", type(exc).__name__)
            raise

def lookup_order(order_id: str) -> dict:
    return {"order_id": order_id, "status": "delivered"}

_transient_failures = {"issue_refund": 1}  # simulate one timeout, then success

def issue_refund(order_id: str) -> dict:
    if _transient_failures["issue_refund"] > 0:
        _transient_failures["issue_refund"] -= 1
        raise TimeoutError("upstream refund API timed out")
    return {"refunded": True}

def run_agent(question: str) -> None:
    with tracer.start_as_current_span("invoke_agent support-agent") as root:
        root.set_attribute("gen_ai.operation.name", "invoke_agent")
        root.set_attribute("gen_ai.provider.name", PROVIDER)
        root.set_attribute("gen_ai.agent.name", "support-agent")

        model_calls = 0
        traced_chat(question)
        model_calls += 1
        traced_tool("lookup_order", lookup_order, order_id="A-1042")
        traced_chat("order found, decide next step")
        model_calls += 1

        for attempt in range(1, 3):
            try:
                traced_tool("issue_refund", issue_refund, order_id="A-1042")
                break
            except TimeoutError:
                root.add_event("tool_retry", {"tool": "issue_refund", "attempt": attempt})

        traced_chat("write the final answer")
        model_calls += 1
        root.set_attribute("app.agent.model_calls", model_calls)  # custom attribute

if __name__ == "__main__":
    run_agent("Where is my order A-1042, and can I get a refund?")

Four notes on the listing:

ERROR. The only addition is error.type, which makes failures groupable by class. Running the listing produces seven spans: one invoke_agent, three chat and three execute_tool.

Three observations follow directly from this data.

chat spans. In this example the model stub dominates run time by construction. In a real system, the split between model time and tool time is exactly what the trace is meant to reveal.

Sampling. Head-based sampling decides before a run starts and therefore discards the rare runs that matter: failures, loops and unusually expensive runs. Tail-based sampling (for example, the OpenTelemetry Collector's tail sampling processor) decides after the run completes. It can keep every run with an error status, a high model-call count or high token usage, plus a small fraction of ordinary runs as a baseline.

Cardinality. Identifiers such as conversation IDs or order numbers belong on spans, not on metric dimensions. Derive metrics from spans with bounded labels: agent name, model, tool name and error.type. The GenAI conventions also define client metrics for token usage and operation duration, which fit this pattern.

Content capture. Prompts and completions are large and often sensitive. Keep capture off by default, enable it per environment behind redaction, and prefer storing a reference to the payload over the payload itself.

Context propagation across agents. When one agent hands work to another over HTTP or a message queue, propagate the W3C trace context in the request headers or message metadata. Otherwise the run fragments into disconnected traces, and the seam, where multi-agent failures tend to occur, becomes the part that cannot be seen.

Specification drift. Parts of the GenAI conventions are still experimental. Keep attribute names in one module so that a rename is a one-line change, and pin the instrumentation version.

Agents that operate production systems, for example to perform root cause analysis, are both consumers and subjects of observability data.

As consumers, their conclusions should reference the telemetry they rely on (topology, metrics, errors), so that an engineer can verify a diagnosis instead of repeating it. As subjects, their own runs need the trace structure above, plus a record of which actions were proposed, approved and executed. Determinism cannot be demanded of the model, but it can be demanded of the system around it: pinned model and prompt versions, typed tools with narrow permissions, and replayable runs.

Bonree ONE's AI Observability applies this model to applications built on LangChain, LangGraph, Dify and OpenClaw, and places the agent data next to the services beneath it. Bonree ONE Sage AI, Bonree's agent operations platform, builds its fault diagnosis on topology, metric and error data of the kind described here.

invoke_agent with execute_tool children. A broader checklist for reviewing agent instrumentation is available at agent-trace-checklist.

Disclosure: I am part of the Bonree team. The code above works with any OTLP-compatible backend.