{"slug": "instrument-an-llm-agent-with-opentelemetry-triplecloud-blog", "title": "Instrument an LLM Agent with OpenTelemetry · TripleCloud Blog", "summary": "TripleCloud Blog published a guide on instrumenting an LLM agent with OpenTelemetry using GenAI semantic conventions, covering zero-code auto-instrumentation, manual spans, token usage, and multi-turn conversations. The conventions, supported by contributors from Amazon, Elastic, Google, IBM, Microsoft, and others, allow vendor-neutral tracing that works with any OTLP backend such as IceGate or Jaeger.", "body_md": "# How to instrument an LLM agent with OpenTelemetry\n\nIf you run an LLM in production, OpenTelemetry can trace it the same way it traces the rest of your stack. This guide shows how to instrument an LLM agent using the GenAI semantic conventions, the shared vocabulary that every major vendor now maps to: the `gen_ai.*`\n\nattributes plus the `chat`\n\n, `execute_tool`\n\nand `invoke_agent`\n\nspans. It covers zero-code auto-instrumentation, manual spans for agent and tool structure, token usage and cost, grouping multi-turn conversations, and where to send the result. None of it ties you to a vendor, so the data stays yours.\n\nEvery step below was run end to end against a local model and a local IceGate, and the runnable recipes are public in our [integrations cookbook](https://github.com/icegatetech/integrations). [IceGate](https://github.com/icegatetech/icegate) is our open-source observability engine: it receives OTLP natively and stores traces as Apache Parquet and Iceberg files on object storage you own. It is where the traces land in this guide. Any other OTLP backend works the same way, so you can follow along with Jaeger or whatever you already run.\n\n[Why trace an LLM agent with OpenTelemetry](#why-trace-an-llm-agent-with-opentelemetry)Why trace an LLM agent with OpenTelemetry\n\nTo ordinary tracing, an LLM call is a black box. You see a slow HTTP request. You do not see which model was called, how many tokens it burned, or whether it looped on a tool. The [OpenTelemetry GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions-genai) fix that by standardizing how LLM operations are recorded: the model, the token counts, the finish reason, and, if you opt in, the full content of prompts, completions and tool calls.\n\nThe reason to bet on them is who is behind them. The working group's [contributors](https://github.com/open-telemetry/semantic-conventions-genai/graphs/contributors) include people from Amazon, Elastic, Google, IBM, Microsoft, Langtrace, OpenLIT, Scorecard and Traceloop, and vendors like [Datadog](https://docs.datadoghq.com/llm_observability/setup/) already consume the conventions natively. When that many implementers agree on a schema before it is even finalized, adopting it early is a small risk.\n\nThe practical payoff is that you instrument once. Emit standard `gen_ai.*`\n\ntelemetry and you can point it at any OTLP backend, then switch backends later without touching the instrumentation.\n\nWhere the spec stands does affect how you pin things. The GenAI conventions are still in [Development status](https://opentelemetry.io/docs/specs/otel/document-status/) and most attributes are experimental, so pin your instrumentation versions and expect occasional breaking changes as the spec settles. They also live in their own repository now, [ open-telemetry/semantic-conventions-genai](https://github.com/open-telemetry/semantic-conventions-genai); the copies still sitting in the main\n\n`semantic-conventions`\n\nrepo are marked as moved. That split doesn't change any of the attribute names below, but it is the repo to watch.[The GenAI semantic conventions in one table](#the-genai-semantic-conventions-in-one-table)The GenAI semantic conventions in one table\n\nAlmost everything you'll see on a [GenAI span](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md) comes from this short vocabulary:\n\n| What | Attribute / span | Example |\n|---|---|---|\n| Operation (also the span name prefix) | `gen_ai.operation.name` |\n`chat` , `execute_tool` , `invoke_agent` , `embeddings` |\n| Provider | `gen_ai.system` (renamed `gen_ai.provider.name` in newer versions) |\n`openai` , `anthropic` , `aws.bedrock` |\n| Requested model | `gen_ai.request.model` |\n`gpt-4o-mini` |\n| Tokens in / out | `gen_ai.usage.input_tokens` , `gen_ai.usage.output_tokens` |\n`412` / `96` |\n| Why it stopped | `gen_ai.response.finish_reasons` |\n`[\"stop\"]` , `[\"tool_calls\"]` |\n| Agent | `gen_ai.agent.name` , `gen_ai.agent.id` |\n`travel-concierge` |\n| Tool | `gen_ai.tool.name` , `gen_ai.tool.type` , `gen_ai.tool.call.id` |\n`get_weather` / `function` |\n| Content (opt-in) | `gen_ai.input.messages` , `gen_ai.output.messages` , `gen_ai.system_instructions` |\nfull prompt / response |\n\nSpan names follow `{operation} {model-or-name}`\n\n, which gives you `chat gpt-4o-mini`\n\n, `execute_tool get_weather`\n\n, `invoke_agent travel-concierge`\n\n. There are also [two client metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md): `gen_ai.client.operation.duration`\n\n(a latency histogram, in seconds) and `gen_ai.client.token.usage`\n\n(a token histogram, split by `gen_ai.token.type`\n\n).\n\nThere is no combined total-tokens attribute. The conventions define input and output only, so if you want a total, sum the two.\n\n[Step 1: Auto-instrument the OpenAI SDK in Python (no code changes)](#step-1-auto-instrument-the-openai-sdk-in-python-no-code-changes)Step 1: Auto-instrument the OpenAI SDK in Python (no code changes)\n\nThis step is Python, and so is the application code in every step after it. The package names, the launcher and the environment-variable handling below are all specific to the Python instrumentation. If you're on Node or TypeScript, three of those details change, and the last section of this step covers them.\n\nThe official OpenTelemetry instrumentation for the OpenAI SDK is [ opentelemetry-instrumentation-openai-v2](https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-openai-v2), part of\n\n`opentelemetry-python-contrib`\n\nand published [on PyPI](https://pypi.org/project/opentelemetry-instrumentation-openai-v2/). Install it alongside the OTLP exporter, pinned:\n\n```\npip install opentelemetry-distro==0.65b0 \\\n            opentelemetry-sdk==1.44.0 \\\n            opentelemetry-exporter-otlp-proto-grpc==1.44.0 \\\n            opentelemetry-instrumentation-openai-v2==2.4b0 \\\n            openai==2.50.0\n```\n\nThese five were [verified working together](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/pyproject.toml) on 2026-07-29 under Python 3.14.5. They are pre-1.0 packages, so re-check the set before you pin it into your own build.\n\nSomething has to be listening on the other end. The endpoint below is `http://localhost:4317`\n\n, so an OTLP receiver has to be running on that port before any of this produces a visible trace. A Jaeger, Grafana Tempo or Aspire dashboard container covers local debugging; an OpenTelemetry Collector, a vendor's agent or IceGate covers the rest, and Step 6 goes into the choice. Point the exporter at a port with nothing behind it and your app keeps working, because the exporter retries and logs the failure instead of raising. What you get is export warnings on stderr and no traces at the far end, which looks exactly like broken instrumentation when the instrumentation is fine.\n\nThat is why the recipe's prerequisites list two processes rather than one: Ollama running locally with the model already pulled, and IceGate listening on `4317`\n\nfor OTLP gRPC. Swap in whichever OTLP backend you already have for the second.\n\nConfigure export and behavior through environment variables, without writing any code:\n\n```\nexport OTEL_SERVICE_NAME=my-llm-app\nexport OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317\nexport OTEL_EXPORTER_OTLP_PROTOCOL=grpc\n\n# Read when the instrumentation loads, so it has to be set BEFORE the process\n# starts. Exporting it from Python is too late. Emits the latest GenAI\n# conventions instead of the frozen v1.30 default.\nexport OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental\n\n# OFF by default. This is an ENUM, not a boolean: NO_CONTENT, SPAN_ONLY,\n# EVENT_ONLY, SPAN_AND_EVENT. Any other value (including `true`) logs a\n# warning and silently falls back to NO_CONTENT. See Step 4 to opt in.\nexport OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT\n```\n\nPort, protocol and installed exporter package all have to agree. `4317`\n\nis the OTLP/gRPC port and pairs with `grpc`\n\n, which is what the pinned `opentelemetry-exporter-otlp-proto-grpc`\n\nspeaks. To export over HTTP instead, move the endpoint to `4318`\n\n, set `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf`\n\n, and install [ opentelemetry-exporter-otlp-proto-http](https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/), which is the step people skip. It is a\n\n[separate distribution](https://opentelemetry.io/docs/languages/python/exporters/), and with only the gRPC package installed the launcher cannot resolve the HTTP exporter and fails at startup.\n\nThen run your app under the OpenTelemetry launcher. Every OpenAI call now produces a `chat`\n\nspan automatically:\n\n```\nopentelemetry-instrument python app.py\n```\n\nWe [verified that on a file importing nothing from OpenTelemetry](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/app_zerocode.py). Zero code changes is not zero dependencies, though. The distro and instrumentation packages above still have to be installed before the launcher has anything to discover.\n\nIf you'd rather wire it up in code, `OpenAIInstrumentor().instrument()`\n\nis the call that patches the SDK:\n\n``` python\nfrom openai import OpenAI\nfrom opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor\n\nOpenAIInstrumentor().instrument()   # every OpenAI call now emits a `chat` span\n\nclient = OpenAI()\nresp = client.chat.completions.create(\n    model=\"gpt-4o-mini\",\n    messages=[{\"role\": \"user\", \"content\": \"Summarize today's incidents.\"}],\n)\n```\n\nThat alone will not export anything. `instrument()`\n\ncreates spans, but it has no opinion about where they go. Under `opentelemetry-instrument`\n\nthe launcher was doing that half of the job, building a `TracerProvider`\n\nwith an OTLP exporter out of the environment variables above. Drop the launcher, run plain `python app.py`\n\n, and the global tracer stays the default no-op: the app runs normally and no span ever leaves the process. Nothing in the logs hints at why, because no exporter was ever built to fail at sending one.\n\nSo replacing the launcher means doing both halves yourself:\n\n``` python\nfrom opentelemetry import trace\nfrom opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter\nfrom opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor\nfrom opentelemetry.sdk.resources import Resource\nfrom opentelemetry.sdk.trace import TracerProvider\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\n\nprovider = TracerProvider(resource=Resource.create({\"service.name\": \"my-llm-app\"}))\nprovider.add_span_processor(\n    BatchSpanProcessor(OTLPSpanExporter(endpoint=\"http://localhost:4317\", insecure=True))\n)\ntrace.set_tracer_provider(provider)   # has to happen before instrument()\n\nOpenAIInstrumentor().instrument()\n\n# ... your app runs here ...\n\nprovider.shutdown()                   # flush before exit\n```\n\nThe `shutdown()`\n\ncall at the end fails the same quiet way when you forget it. `BatchSpanProcessor`\n\nqueues spans and flushes on a timer, so a short script that does its work and exits takes the pending batch with it, and you are back to an empty backend. Our recipe's [ telemetry.py](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/telemetry.py) handles both halves in one place and returns the provider so the caller can shut it down.\n\nOne constraint does not move into code at all. `OTEL_SEMCONV_STABILITY_OPT_IN`\n\nis read when the instrumentation module loads, so setting `os.environ[...]`\n\nabove `instrument()`\n\nis already too late. You silently get the v1.30 names back (`gen_ai.system`\n\ninstead of `gen_ai.provider.name`\n\n) and nothing raises to tell you. Export it in your shell or `.env`\n\nbefore the process starts. That same `telemetry.py`\n\nexits loudly on a missing opt-in instead of quietly emitting the old names.\n\nBy default the instrumentation records metadata only (model, token counts, finish reason, duration) and leaves message content out, which is the right default for production. Step 4 covers turning content on safely.\n\n[Which package to install](#which-package-to-install)Which package to install\n\nUse `opentelemetry-instrumentation-openai-v2`\n\n, the official OTel implementation. The older `opentelemetry-instrumentation-openai`\n\nis Traceloop's community package (OpenLLMetry). It's still maintained, but it isn't the standard. For Anthropic, Bedrock, Vertex and others you have a choice: their OpenAI-compatible endpoints, a community or framework instrumentor (OpenLLMetry, or framework-native ones for LangChain, LlamaIndex, the OpenAI Agents SDK, CrewAI), or the dedicated agents instrumentation. All of them emit the same `gen_ai.*`\n\nattributes, so everything below is identical regardless of provider.\n\n[Pointing at an OpenAI-compatible endpoint](#pointing-at-an-openai-compatible-endpoint)Pointing at an OpenAI-compatible endpoint\n\nIn practice the SDK requires two things, and the model name has to move with them:\n\n``` python\nimport os\nfrom openai import OpenAI\n\nclient = OpenAI(\n    base_url=\"http://localhost:11434/v1\",  # any OpenAI-compatible endpoint\n    api_key=\"ollama\",                      # required by the SDK; ignored by the server\n)\n\nresp = client.chat.completions.create(\n    model=os.environ.get(\"OLLAMA_MODEL\", \"gemma4:12b-mlx\"),\n    messages=[{\"role\": \"user\", \"content\": \"Summarize today's incidents.\"}],\n)\n```\n\nChange only the `base_url`\n\nand you get a model-not-found error instead of a trace. A local Ollama serves exactly the models you have pulled onto that machine, and `gpt-4o-mini`\n\nis not one of them, so the model name has to change along with the endpoint. Our recipe reads it from `OLLAMA_MODEL`\n\n, which is why its prerequisites start with `ollama pull gemma4:12b-mlx`\n\n.\n\nThe emitted provider value describes the SDK and protocol surface, not the backend actually serving the request. Point the OpenAI SDK at Ollama and you get `gen_ai.provider.name = \"openai\"`\n\n, not `\"ollama\"`\n\n. That matters if you group cost or latency by provider.\n\n[If you're on Node or TypeScript](#if-youre-on-node-or-typescript)If you're on Node or TypeScript\n\nThe steps above are Python. The JS path differs in three ways that the rest of this guide doesn't cover:\n\ndeclares`@opentelemetry/instrumentation-openai`\n\n`supportedVersions: \">=4.19.0 <7\"`\n\n(checked against 0.19.0). With`openai`\n\n7.x installed it silently never patches: your app runs fine and emits no spans at all. Pin`openai`\n\nto 6.x.- It has no\n`OTEL_SEMCONV_STABILITY_OPT_IN`\n\nsupport, so its Chat Completions path emits the deprecated`gen_ai.system`\n\n. The Step 2 advice below about`gen_ai.provider.name`\n\ncan't be followed in JS today. - Under Node ESM, calling\n`registerInstrumentations()`\n\nat runtime is not enough. The loader hook has to be registered at process start:`node --import tsx --experimental-loader=@opentelemetry/instrumentation/hook.mjs src/index.ts`\n\nOur [TypeScript recipe](https://github.com/icegatetech/integrations/blob/main/typescript/openai-ollama/package.json) carries both fixes: the `openai`\n\n6.49.0 pin, and the loader flag in its `start`\n\nscript.\n\n[Step 2: Add agent and tool spans with invoke_agent and execute_tool](#step-2-add-agent-and-tool-spans-with-invoke_agent-and-execute_tool)Step 2: Add agent and tool spans with invoke_agent and execute_tool\n\nAuto-instrumentation gives you a flat list of `chat`\n\nspans. What it can't know is your agent's structure: which calls belong to which task, and when a tool ran. You add that by wrapping the run loop in an [ invoke_agent span](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md) and each tool call in an\n\n`execute_tool`\n\nspan. The auto-instrumented `chat`\n\nspans then nest underneath on their own.The sketch below is illustrative pseudocode: `call_model`\n\n, `dispatch_tool`\n\nand the shape of `plan`\n\nare yours to supply. For a complete runnable version, with a real client call, real tool dispatch and real control flow around the tool-call decision, see [ agent.py](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/agent.py) in the cookbook. The one trap it has to handle comes up after the trace diagram.\n\n``` python\nfrom opentelemetry import trace\nfrom opentelemetry.instrumentation.openai_v2 import OpenAIInstrumentor\n\nOpenAIInstrumentor().instrument()\ntracer = trace.get_tracer(\"travel.concierge\")\n\nAGENT = \"travel-concierge\"\nAGENT_ID = \"travel-concierge-001\"\n\ndef run_agent(user_msg):\n    with tracer.start_as_current_span(\n        f\"invoke_agent {AGENT}\",\n        kind=trace.SpanKind.INTERNAL,\n        attributes={\n            \"gen_ai.operation.name\": \"invoke_agent\",\n            \"gen_ai.agent.name\": AGENT,\n            \"gen_ai.agent.id\": AGENT_ID,\n            \"gen_ai.provider.name\": \"openai\",\n        },\n    ):\n        # 1) model decides what to do; auto-instrumented as a child `chat` span\n        plan = call_model(user_msg)\n\n        # 2) wrap each tool call in its own `execute_tool` span\n        for call in plan.tool_calls:\n            with tracer.start_as_current_span(\n                f\"execute_tool {call.name}\",\n                attributes={\n                    \"gen_ai.operation.name\": \"execute_tool\",\n                    \"gen_ai.tool.name\": call.name,\n                    \"gen_ai.tool.type\": \"function\",\n                    \"gen_ai.tool.call.id\": call.id,\n                },\n            ):\n                call.result = dispatch_tool(call.name, call.args)\n\n        # 3) model writes the final answer, another child `chat` span\n        return call_model(user_msg, tool_results=plan.tool_calls)\n```\n\nNote `gen_ai.provider.name`\n\nrather than `gen_ai.system`\n\n. Once you have opted into `gen_ai_latest_experimental`\n\nin Step 1, that is the current name for this attribute; `gen_ai.system`\n\nis the name that convention deprecated. Most backends still accept the old spelling through a deprecation fallback, so nothing breaks if you use it, but it contradicts the opt-in you just set.\n\n`gen_ai.agent.id`\n\nand `gen_ai.tool.call.id`\n\nare worth setting too. The provider already returns a tool-call id, and backends with typed columns for these can only populate them if you emit them.\n\nThe result is a trace with the same shape as the run:\n\nWith that structure in place you can tell where a 45-second response went: into a slow tool, into a retry loop, or into the model call itself.\n\nThe trap is `tool_choice`\n\n. Set it to `\"required\"`\n\non the planning call if you want the tool step to be deterministic, and make sure you don't set it on the final call. Left on, it forces the model to request another tool and the loop never terminates.\n\n[Step 3: From token usage to LLM cost per request](#step-3-from-token-usage-to-llm-cost-per-request)Step 3: From token usage to LLM cost per request\n\nThe instrumentation records `gen_ai.usage.input_tokens`\n\nand `gen_ai.usage.output_tokens`\n\non every `chat`\n\nspan, so per-request cost is those counts multiplied by your model's price per token. The `gen_ai.client.token.usage`\n\nmetric lets you spot token-hungry prompts before they reach production, and `gen_ai.response.finish_reasons`\n\ntells you when a call ended on `tool_calls`\n\nrather than `stop`\n\n, which is how you catch runaway tool loops.\n\nThe token counts land automatically. Currency does not, because that needs a price table, and a fresh deployment may not have one. On IceGate the built-in `prices`\n\ntable is filled by a crawler that ships disabled, so cost columns read `NULL`\n\nuntil you turn it on. Our [cost query](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/queries/cost_per_run.sql) is the one a real deployment should run, and on a stock install it returns true token counts next to `NULL`\n\ncosts.\n\nTo get real rates, turn the crawler on with `maintain.pricing.enabled`\n\nin IceGate's Helm values. It ships off because it is the one maintain workload that makes outbound internet calls, and granting the pod egress should be a deliberate choice rather than a default someone inherits. The shipped source list already covers OpenRouter and LiteLLM, so there is usually nothing else to configure, but `prices`\n\nhas to exist before the first crawl: the crawler appends to that table and will not create it. Run `migrate`\n\nfirst.\n\nIf you only need the arithmetic, skip the table and supply the rates inline. Our [worked example](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/queries/cost_per_run_worked_example.sql) runs the same calculation against a `WITH rates AS (...)`\n\nCTE, so it returns real numbers on a stock install with an empty `prices`\n\ntable. That is also the query for a local model, which has no upstream rate card at all: the token counts are genuine measurements, and you are pricing them as if a paid model had served them.\n\nThese `chat`\n\nspans are also big, which matters later. Prompts, completions and tool payloads make a single `chat`\n\nspan roughly fifty times the size of an ordinary HTTP or database span, the ~20 KB per span our own pricing model assumes. That size gap is why volume-priced observability gets expensive so fast, and we work the numbers through in our companion piece on [why LLM observability bills explode at scale](/posts/the-ai-tax-on-telemetry).\n\n[Step 4: Capture prompts and responses safely without leaking PII](#step-4-capture-prompts-and-responses-safely-without-leaking-pii)Step 4: Capture prompts and responses safely without leaking PII\n\nContent capture is off by default, because prompts and responses routinely contain PII. Turn it on with the environment variable from Step 1:\n\n```\nexport OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=SPAN_ONLY\n```\n\nThe value is an enum, not a boolean: `NO_CONTENT`\n\n, `SPAN_ONLY`\n\n, `EVENT_ONLY`\n\nor `SPAN_AND_EVENT`\n\n. `SPAN_ONLY`\n\ngives you the on-span attributes described below. `true`\n\nand `false`\n\nbelong to the older pre-opt-in path and are rejected once `gen_ai_latest_experimental`\n\nis set. The library logs a warning and falls back to `NO_CONTENT`\n\n, so you get no content at all and one log line explaining why.\n\nWith `gen_ai_latest_experimental`\n\nenabled, the instrumentation records content as structured span attributes (`gen_ai.input.messages`\n\n, `gen_ai.output.messages`\n\n, `gen_ai.system_instructions`\n\n), so you can replay the exact conversation when debugging. In legacy mode it emits the same content as log events instead.\n\nIn production these attributes are both large and sensitive. Redact at the OpenTelemetry Collector before the data leaves your network, or keep full content only in storage you control.\n\n[Step 5: Group multi-turn conversations with gen_ai.conversation.id](#step-5-group-multi-turn-conversations-with-gen_aiconversationid)Step 5: Group multi-turn conversations with gen_ai.conversation.id\n\nEverything so far covers a single turn. A real agent fields many turns from one user over time, and each turn is its own trace with its own root span. Without a shared key, a backend shows you a pile of unrelated traces that happen to share an agent name, instead of a conversation you can read end to end.\n\nThe attribute for this is [ gen_ai.conversation.id](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md), which the spec defines as a \"unique identifier for a conversation (session, thread)\". It is a span attribute, not a resource attribute. OpenInference's equivalent is\n\n`session.id`\n\n. IceGate populates its typed `operations.conversation_id`\n\ncolumn from it, and that column exists to group a conversation's operations across traces.The obvious approach does not work. Setting the id on the `invoke_agent`\n\nspan you already create in Step 2 leaves it on that one span, because span attributes don't inherit. Every auto-instrumented `chat`\n\nspan nested underneath stays empty, which defeats the grouping exactly where it matters: the model calls are the spans you most want to line up across turns.\n\nA resource attribute would cover every span for free, and it is still wrong. Resources are process-wide and immutable once the `TracerProvider`\n\nis built; conversations are neither. One process serves many conversations, sequentially or concurrently, and one conversation can span many processes. Bake a conversation id into the resource and it breaks the first time either of those is true.\n\nWhat works is a span processor. Register one whose `on_start`\n\nhook (`onStart`\n\nin Node) reads the active conversation id and stamps it on each span as the span opens. Because `on_start`\n\nruns while the span is still live, `span.set_attribute()`\n\nhere is ordinary public API rather than a reach into private SDK internals or into a span that has already ended. Register it as the outermost processor on the `TracerProvider`\n\nand coverage becomes uniform, since every span the process creates passes through the same hook, whether your own code created it or an auto-instrumentation did, deep inside a library call.\n\nThe active id itself comes from a `ContextVar`\n\nin Python or `AsyncLocalStorage`\n\nin Node. Both are the standard mechanism in their language for ambient per-call context that survives async boundaries.\n\nUnlike the sketch in Step 2, this is real working code, taken from [ conversation.py](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/conversation.py) in the cookbook (the Node port is\n\n[):](https://github.com/icegatetech/integrations/blob/main/typescript/openai-ollama/src/conversation.ts)\n\n`conversation.ts`\n\n``` python\nfrom contextlib import contextmanager\nfrom contextvars import ContextVar\n\nfrom opentelemetry.sdk.trace import SpanProcessor\n\n_current_conversation_id: ContextVar[str | None] = ContextVar(\n    \"icegate_conversation_id\", default=None\n)\n\ndef current_id() -> str | None:\n    \"\"\"The active conversation id, or None outside any `conversation()` block.\"\"\"\n    return _current_conversation_id.get()\n\n@contextmanager\ndef conversation(conversation_id: str | None):\n    \"\"\"Make `conversation_id` the active conversation for the block.\"\"\"\n    token = _current_conversation_id.set(conversation_id)\n    try:\n        yield conversation_id\n    finally:\n        _current_conversation_id.reset(token)   # restores on exception too\n\nclass ConversationSpanProcessor(SpanProcessor):\n    \"\"\"Stamps `gen_ai.conversation.id` on every span started while a conversation is active.\"\"\"\n\n    def __init__(self, inner: SpanProcessor):\n        self._inner = inner\n\n    def on_start(self, span, parent_context=None):\n        conversation_id = current_id()          # reads the ContextVar above\n        if conversation_id is not None:\n            span.set_attribute(\"gen_ai.conversation.id\", conversation_id)\n        self._inner.on_start(span, parent_context=parent_context)\n\n    # The wrapper owns the whole SpanProcessor contract, so the remaining\n    # hooks have to reach `inner`. Drop these and batching never flushes.\n    def on_end(self, span):\n        self._inner.on_end(span)\n\n    def shutdown(self):\n        self._inner.shutdown()\n\n    def force_flush(self, timeout_millis=30000):\n        return self._inner.force_flush(timeout_millis=timeout_millis)\n```\n\nWrap the work in `conversation(\"conv-bc3ceb84\")`\n\nand every span opened inside the block carries the id, including the ones auto-instrumentation creates deep inside a library call.\n\nTwo turns then share one id while staying two separate traces:\n\nA real two-turn run, queried straight out of IceGate's `operations`\n\ntable, gives 8 spans with two distinct `trace_id`\n\ns (`d1161d75…`\n\nand `e0fd6bbe…`\n\n) and one shared `conversation_id`\n\n, `conv-bc3ceb84…`\n\n, on every one of them. Four of those spans are the auto-instrumented `chat`\n\nspans, which the recipe never sets `gen_ai.conversation.id`\n\non directly. The span processor is what reached them, because application code never touches those spans at all.\n\nThe recipe [runs the agent twice](https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/recipe.py) on purpose. A single turn is a single trace, which would demonstrate nothing about grouping across traces.\n\n[Step 6: Where to send your gen_ai telemetry](#step-6-where-to-send-your-gen_ai-telemetry)Step 6: Where to send your gen_ai telemetry\n\nAny OTLP-compatible backend can receive this telemetry. For local debugging, a Jaeger, Grafana Tempo or Aspire dashboard container works. In production, the vendors that consume the GenAI conventions (Datadog, SigNoz, Uptrace and others) all accept the same data. That is what instrumenting to the standard buys you. Changing where the data goes is a one-line change instead of a re-instrumentation project.\n\nThe cost problem from Step 3 gets resolved at the same layer. Point the same OTLP exporter at IceGate, our open-source observability engine, and those heavy `chat`\n\nspans land in open Parquet and Iceberg files on your own object storage, queryable with plain SQL. Because retention sits on cheap object storage rather than a proprietary indexed store, you can keep full-fidelity traces, prompts and tool calls included, instead of sampling them away. Nothing about your instrumentation changes: you keep the open OpenTelemetry data you just emitted, and it stays yours. Our companion piece on [the observability data lake](/posts/observability-ingestion-on-s3) covers that architecture in depth.\n\nIceGate is Apache-2.0 and speaks OTLP natively. [Star it on GitHub](https://github.com/icegatetech/icegate), point your collector at it, and watch your agent's traces land in open formats. Every step above has a runnable, verified recipe in the [integrations cookbook](https://github.com/icegatetech/integrations), in both Python and TypeScript.\n\n[FAQ](#faq)FAQ\n\n[What are the OpenTelemetry GenAI semantic conventions?](#what-are-the-opentelemetry-genai-semantic-conventions)What are the OpenTelemetry GenAI semantic conventions?\n\nA standard set of span names, attributes (`gen_ai.*`\n\n), metrics and events for recording generative-AI operations: the model, tokens, finish reason, agent and tool structure, and optional prompt and response content. Emitting them keeps AI telemetry consistent across frameworks and vendors.\n\n[How do I instrument an LLM with OpenTelemetry without rewriting my app?](#how-do-i-instrument-an-llm-with-opentelemetry-without-rewriting-my-app)How do I instrument an LLM with OpenTelemetry without rewriting my app?\n\nInstall the provider's OTel instrumentation (for OpenAI, `opentelemetry-instrumentation-openai-v2`\n\n), set the OTLP environment variables, and run under `opentelemetry-instrument`\n\n. Every model call is traced automatically; add manual `invoke_agent`\n\nand `execute_tool`\n\nspans only where you want agent and tool structure.\n\n[Which OTLP port and protocol should I use?](#which-otlp-port-and-protocol-should-i-use)Which OTLP port and protocol should I use?\n\nMatch all three: `4317`\n\nwith `OTEL_EXPORTER_OTLP_PROTOCOL=grpc`\n\nand the `opentelemetry-exporter-otlp-proto-grpc`\n\npackage, or `4318`\n\nwith `http/protobuf`\n\nand `opentelemetry-exporter-otlp-proto-http`\n\n. The two exporters are separate distributions, so selecting a protocol whose package isn't installed makes the launcher fail at startup.\n\n[Which gen_ai attributes matter most?](#which-gen_ai-attributes-matter-most)Which gen_ai attributes matter most?\n\n`gen_ai.request.model`\n\n, `gen_ai.usage.input_tokens`\n\n/ `output_tokens`\n\n(for cost), and `gen_ai.response.finish_reasons`\n\n(for loop and error detection). Add `gen_ai.agent.name`\n\nand `gen_ai.tool.name`\n\nto give traces structure.\n\n[Which convention version am I emitting?](#which-convention-version-am-i-emitting)Which convention version am I emitting?\n\nBy default the OpenAI-v2 instrumentation follows conventions around v1.30 and won't record newer details. Set `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`\n\nto emit the latest version, including on-attribute content capture. Set it in the environment before the process starts: the instrumentation reads it at import time, so setting it from Python is too late. The provider attribute `gen_ai.system`\n\nis also being renamed to `gen_ai.provider.name`\n\nin newer versions, so if you opt in, use the new name in your manual spans too.\n\n[Do I have to capture prompt content?](#do-i-have-to-capture-prompt-content)Do I have to capture prompt content?\n\nNo. It's off by default, and only metadata is recorded. Enable it deliberately with `SPAN_ONLY`\n\n(not `true`\n\n, since the setting is an enum), redact at the Collector, and prefer keeping full content in storage you control.\n\n[Does this work for Anthropic, Bedrock, or my agent framework?](#does-this-work-for-anthropic,-bedrock,-or-my-agent-framework)Does this work for Anthropic, Bedrock, or my agent framework?\n\nYes. Use OpenAI-compatible endpoints, a community or framework instrumentor, or the dedicated agents instrumentation. All of them emit the same `gen_ai.*`\n\nattributes, so your dashboards and queries don't change when you switch providers. Language is a different matter: see the Node and TypeScript note in Step 1.\n\n[How do I group a multi-turn conversation into one view?](#how-do-i-group-a-multi-turn-conversation-into-one-view)How do I group a multi-turn conversation into one view?\n\nStamp `gen_ai.conversation.id`\n\non every span, not just the agent span. Span attributes do not inherit, so an id set only on `invoke_agent`\n\nleaves every auto-instrumented `chat`\n\nspan without it. The pattern that covers all of them is a span processor that reads the active conversation id from a `ContextVar`\n\n(Python) or `AsyncLocalStorage`\n\n(Node) in its `on_start`\n\nhook, as in Step 5.\n\n[Trademarks and disclaimer](#trademarks-and-disclaimer)Trademarks and disclaimer\n\n**Trademarks.** OpenTelemetry and Jaeger are trademarks of the Cloud Native Computing Foundation (CNCF) / The Linux Foundation. Grafana® and Tempo® are trademarks of Raintank, Inc. (Grafana Labs). Apache®, Apache Iceberg™, and Apache Parquet™ are trademarks of The Apache Software Foundation. OpenAI® is a trademark of OpenAI OpCo, LLC. Anthropic® is a trademark of Anthropic PBC. Amazon S3®, Amazon Bedrock®, and AWS® are trademarks of Amazon Technologies, Inc. Google® and Vertex AI® are trademarks of Google LLC. Microsoft® is a trademark of Microsoft Corporation. IBM® is a trademark of International Business Machines Corporation. Elasticsearch® is a trademark of Elasticsearch B.V. / Elastic N.V. Datadog® is a trademark of Datadog, Inc. Docker® is a trademark of Docker, Inc. Python® is a trademark of the Python Software Foundation. Node.js® is a trademark of the OpenJS Foundation. SigNoz, Uptrace, Ollama, LangChain, LlamaIndex, CrewAI, Traceloop / OpenLLMetry, OpenLIT, Langtrace, and Scorecard are designations of their respective owners. All other product names, logos, and brands are the property of their respective owners. Names are used solely for identification and do not imply affiliation, sponsorship, or endorsement by the trademark holders.\n\n**Disclaimer.** IceGate and TripleCloud are not affiliated with or endorsed by the Cloud Native Computing Foundation (CNCF), The Apache Software Foundation, OpenAI, Anthropic, Amazon Web Services, Google, Microsoft, IBM, Elastic, Grafana Labs, Datadog, or any of the other companies mentioned here. The instructions and version pins above reflect the packages and configurations described, verified on the dates stated, and may not account for newer releases; the OpenTelemetry GenAI semantic conventions are still experimental and subject to breaking change. Corrections and clarifications are welcome, and we will fix any inaccuracies.", "url": "https://wpnews.pro/news/instrument-an-llm-agent-with-opentelemetry-triplecloud-blog", "canonical_source": "https://blog.triplecloud.tech/posts/instrument-llm-agent-opentelemetry", "published_at": "2026-07-30 20:10:53+00:00", "updated_at": "2026-07-30 20:22:10.807119+00:00", "lang": "en", "topics": ["artificial-intelligence", "large-language-models", "ai-agents", "developer-tools"], "entities": ["TripleCloud Blog", "OpenTelemetry", "IceGate", "Amazon", "Elastic", "Google", "IBM", "Microsoft"], "alternates": {"html": "https://wpnews.pro/news/instrument-an-llm-agent-with-opentelemetry-triplecloud-blog", "markdown": "https://wpnews.pro/news/instrument-an-llm-agent-with-opentelemetry-triplecloud-blog.md", "text": "https://wpnews.pro/news/instrument-an-llm-agent-with-opentelemetry-triplecloud-blog.txt", "jsonld": "https://wpnews.pro/news/instrument-an-llm-agent-with-opentelemetry-triplecloud-blog.jsonld"}}