{"slug": "tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai", "title": "Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability", "summary": "A developer designed a run-centric trace model for LLM and agentic AI observability built on OpenTelemetry's GenAI semantic conventions (gen_ai.*), using one trace per run with three span types: invoke_agent, chat, and execute_tool. The design attaches token usage to every model-call span and error status to every failed tool call, which the author argues is enough to answer which path an agent took, what it cost, and what changed during an incident. A verified example agent with stubbed model and tool clients runs without an API key, and the writeup covers production trade-offs including sampling, cardinality, content capture, and context propagation across agents.", "body_md": "\n\n```\ntitle: \"Tracing AI Agents with OpenTelemetry: A Design for LLM and Agentic AI Observability\"\npublished: false\ndescription: \"A run-centric trace model for LLM and agentic AI observability with OpenTelemetry GenAI conventions, with measurements from a verified example.\"\ntags: opentelemetry, observability, devops, bonree\n```\n\nLLM applications and agentic AI operations share a property that classic APM was not designed for: the execution path is decided at runtime by a model. A request that returns HTTP 200 may have looped, retried a failing tool, or consumed several times the usual number of tokens, and none of that is visible from the response.\n\nThis article argues for a run-centric trace model, implements it with the OpenTelemetry GenAI semantic conventions (`gen_ai.*`), and reports what the trace contains when an example agent is executed. It then covers the production trade-offs that decide whether such tracing survives real traffic: sampling, cardinality, content capture and context propagation across agents.\n\n**Conclusion.** One trace per run, three span types (`invoke_agent`, `chat`, `execute_tool`), token usage on every model-call span and error status on every failed tool call are sufficient to answer three of the four questions that recur in agent incidents: which path was taken, what it cost, and what was changed. The fourth, whether the answer was correct, needs an evaluation signal that tracing cannot supply.\n\nFour properties distinguish an agent run from a request:\n\nThe consequence is that the entity to observe is the run, and its internal structure matters more than its endpoint.\n\n| Span name | `gen_ai.operation.name` | Represents | \n|---|---|---|\n| `invoke_agent {agent}` | `invoke_agent` | The whole run | \n| `chat {model}` | `chat` | One model request | \n| `execute_tool {tool}` | `execute_tool` | One tool invocation | \n\nThree design choices are worth stating explicitly.\n\n`app.*`), so that a later addition to the specification cannot collide with it.\nThe listing is complete and runs without an API key. The model client and the tools are stubs, and the refund tool is configured to fail once and then succeed, so every run is identical.\n\n``` python\nimport time\n\nfrom opentelemetry import trace\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor, ConsoleSpanExporter\n\nprovider = TracerProvider(resource=Resource.create({\"service.name\": \"support-agent\"}))\nprovider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter()))\ntrace.set_tracer_provider(provider)\ntracer = trace.get_tracer(\"support-agent\")\n\nPROVIDER = \"example-provider\"\nMODEL = \"example-model\"\n\ndef call_llm(prompt: str) -> dict:\n    \"\"\"Stand-in for a real model client.\"\"\"\n    time.sleep(0.2)\n    return {\n        \"text\": \"...\",\n        \"input_tokens\": len(prompt.split()) * 2,\n        \"output_tokens\": 48,\n        \"finish_reason\": \"stop\",\n    }\n\ndef traced_chat(prompt: str) -> dict:\n    with tracer.start_as_current_span(f\"chat {MODEL}\") as span:\n        span.set_attribute(\"gen_ai.operation.name\", \"chat\")\n        span.set_attribute(\"gen_ai.provider.name\", PROVIDER)\n        span.set_attribute(\"gen_ai.request.model\", MODEL)\n        resp = call_llm(prompt)\n        span.set_attribute(\"gen_ai.usage.input_tokens\", resp[\"input_tokens\"])\n        span.set_attribute(\"gen_ai.usage.output_tokens\", resp[\"output_tokens\"])\n        span.set_attribute(\"gen_ai.response.finish_reasons\", [resp[\"finish_reason\"]])\n        return resp\n\ndef traced_tool(name: str, fn, **args):\n    # An exception raised inside the span is recorded and the span status is set\n    # to ERROR automatically by the SDK; we only add the semantic-convention\n    # error.type attribute so failures can be grouped by class.\n    with tracer.start_as_current_span(f\"execute_tool {name}\") as span:\n        span.set_attribute(\"gen_ai.operation.name\", \"execute_tool\")\n        span.set_attribute(\"gen_ai.tool.name\", name)\n        try:\n            return fn(**args)\n        except Exception as exc:\n            span.set_attribute(\"error.type\", type(exc).__name__)\n            raise\n\ndef lookup_order(order_id: str) -> dict:\n    return {\"order_id\": order_id, \"status\": \"delivered\"}\n\n_transient_failures = {\"issue_refund\": 1}  # simulate one timeout, then success\n\ndef issue_refund(order_id: str) -> dict:\n    if _transient_failures[\"issue_refund\"] > 0:\n        _transient_failures[\"issue_refund\"] -= 1\n        raise TimeoutError(\"upstream refund API timed out\")\n    return {\"refunded\": True}\n\ndef run_agent(question: str) -> None:\n    with tracer.start_as_current_span(\"invoke_agent support-agent\") as root:\n        root.set_attribute(\"gen_ai.operation.name\", \"invoke_agent\")\n        root.set_attribute(\"gen_ai.provider.name\", PROVIDER)\n        root.set_attribute(\"gen_ai.agent.name\", \"support-agent\")\n\n        model_calls = 0\n        traced_chat(question)\n        model_calls += 1\n        traced_tool(\"lookup_order\", lookup_order, order_id=\"A-1042\")\n        traced_chat(\"order found, decide next step\")\n        model_calls += 1\n\n        for attempt in range(1, 3):\n            try:\n                traced_tool(\"issue_refund\", issue_refund, order_id=\"A-1042\")\n                break\n            except TimeoutError:\n                root.add_event(\"tool_retry\", {\"tool\": \"issue_refund\", \"attempt\": attempt})\n\n        traced_chat(\"write the final answer\")\n        model_calls += 1\n        root.set_attribute(\"app.agent.model_calls\", model_calls)  # custom attribute\n\nif __name__ == \"__main__\":\n    run_agent(\"Where is my order A-1042, and can I get a refund?\")\n```\n\nFour notes on the listing:\n\n`ERROR`. The only addition is `error.type`, which makes failures groupable by class.\nRunning the listing produces seven spans: one `invoke_agent`, three `chat` and three `execute_tool`.\n\n| Span | Status | Notes | \n|---|---|---|\n| `chat example-model` (1st) | unset | 22 input / 48 output tokens | \n| `execute_tool lookup_order` | unset |  | \n| `chat example-model` (2nd) | unset | 10 input / 48 output tokens | \n| `execute_tool issue_refund` (1st) | **ERROR** | `error.type = TimeoutError` , exception event recorded | \n| `execute_tool issue_refund` (2nd) | unset | The retry succeeded | \n| `chat example-model` (3rd) | unset | 8 input / 48 output tokens | \n| `invoke_agent support-agent` | unset | `tool_retry` event;`app.agent.model_calls = 3` | \n\nThree observations follow directly from this data.\n\n`chat` spans.\nIn this example the model stub dominates run time by construction. In a real system, the split between model time and tool time is exactly what the trace is meant to reveal.\n\n**Sampling.** Head-based sampling decides before a run starts and therefore discards the rare runs that matter: failures, loops and unusually expensive runs. Tail-based sampling (for example, the OpenTelemetry Collector's tail sampling processor) decides after the run completes. It can keep every run with an error status, a high model-call count or high token usage, plus a small fraction of ordinary runs as a baseline.\n\n**Cardinality.** Identifiers such as conversation IDs or order numbers belong on spans, not on metric dimensions. Derive metrics from spans with bounded labels: agent name, model, tool name and `error.type`. The GenAI conventions also define client metrics for token usage and operation duration, which fit this pattern.\n\n**Content capture.** Prompts and completions are large and often sensitive. Keep capture off by default, enable it per environment behind redaction, and prefer storing a reference to the payload over the payload itself.\n\n**Context propagation across agents.** When one agent hands work to another over HTTP or a message queue, propagate the W3C trace context in the request headers or message metadata. Otherwise the run fragments into disconnected traces, and the seam, where multi-agent failures tend to occur, becomes the part that cannot be seen.\n\n**Specification drift.** Parts of the GenAI conventions are still experimental. Keep attribute names in one module so that a rename is a one-line change, and pin the instrumentation version.\n\nAgents that operate production systems, for example to perform root cause analysis, are both consumers and subjects of observability data.\n\nAs consumers, their conclusions should reference the telemetry they rely on (topology, metrics, errors), so that an engineer can verify a diagnosis instead of repeating it. As subjects, their own runs need the trace structure above, plus a record of which actions were proposed, approved and executed. Determinism cannot be demanded of the model, but it can be demanded of the system around it: pinned model and prompt versions, typed tools with narrow permissions, and replayable runs.\n\nBonree ONE's AI Observability applies this model to applications built on LangChain, LangGraph, Dify and OpenClaw, and places the agent data next to the services beneath it. Bonree ONE Sage AI, Bonree's agent operations platform, builds its fault diagnosis on topology, metric and error data of the kind described here.\n\n`invoke_agent` with `execute_tool` children.\nA broader checklist for reviewing agent instrumentation is available at [agent-trace-checklist](https://github.com/Bonree-official/Technology-Blog/blob/main/Agentic-AI-Operations-Evidence-Carrying-Handoffs.md).\n\n*Disclosure: I am part of the Bonree team. The code above works with any OTLP-compatible backend.*", "url": "https://wpnews.pro/news/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai", "canonical_source": "https://dev.to/bonree-js/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai-observability-1cjp", "published_at": "2026-10-30 04:10:22+00:00", "updated_at": "2026-10-30 04:19:34.325865+00:00", "lang": "en", "topics": ["ai-agents", "large-language-models", "mlops", "developer-tools", "ai-infrastructure"], "entities": ["OpenTelemetry", "Bonree"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai", "markdown": "https://wpnews.pro/news/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai.md", "text": "https://wpnews.pro/news/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai.txt", "jsonld": "https://wpnews.pro/news/tracing-ai-agents-with-opentelemetry-a-design-for-llm-and-agentic-ai.jsonld"}}