Instrument an LLM Agent with OpenTelemetry · TripleCloud Blog

TripleCloud Blog published a guide on instrumenting an LLM agent with OpenTelemetry using GenAI semantic conventions, covering zero-code auto-instrumentation, manual spans, token usage, and multi-turn conversations. The conventions, supported by contributors from Amazon, Elastic, Google, IBM, Microsoft, and others, allow vendor-neutral tracing that works with any OTLP backend such as IceGate or Jaeger.

How to instrument an LLM agent with OpenTelemetry If you run an LLM in production, OpenTelemetry can trace it the same way it traces the rest of your stack. This guide shows how to instrument an LLM agent using the GenAI semantic conventions, the shared vocabulary that every major vendor now maps to: the gen ai. attributes plus the chat , execute tool and invoke agent spans. It covers zero-code auto-instrumentation, manual spans for agent and tool structure, token usage and cost, grouping multi-turn conversations, and where to send the result. None of it ties you to a vendor, so the data stays yours. Every step below was run end to end against a local model and a local IceGate, and the runnable recipes are public in our integrations cookbook https://github.com/icegatetech/integrations . IceGate https://github.com/icegatetech/icegate is our open-source observability engine: it receives OTLP natively and stores traces as Apache Parquet and Iceberg files on object storage you own. It is where the traces land in this guide. Any other OTLP backend works the same way, so you can follow along with Jaeger or whatever you already run. Why trace an LLM agent with OpenTelemetry why-trace-an-llm-agent-with-opentelemetry Why trace an LLM agent with OpenTelemetry To ordinary tracing, an LLM call is a black box. You see a slow HTTP request. You do not see which model was called, how many tokens it burned, or whether it looped on a tool. The OpenTelemetry GenAI semantic conventions https://github.com/open-telemetry/semantic-conventions-genai fix that by standardizing how LLM operations are recorded: the model, the token counts, the finish reason, and, if you opt in, the full content of prompts, completions and tool calls. The reason to bet on them is who is behind them. The working group's contributors https://github.com/open-telemetry/semantic-conventions-genai/graphs/contributors include people from Amazon, Elastic, Google, IBM, Microsoft, Langtrace, OpenLIT, Scorecard and Traceloop, and vendors like Datadog https://docs.datadoghq.com/llm observability/setup/ already consume the conventions natively. When that many implementers agree on a schema before it is even finalized, adopting it early is a small risk. The practical payoff is that you instrument once. Emit standard gen ai. telemetry and you can point it at any OTLP backend, then switch backends later without touching the instrumentation. Where the spec stands does affect how you pin things. The GenAI conventions are still in Development status https://opentelemetry.io/docs/specs/otel/document-status/ and most attributes are experimental, so pin your instrumentation versions and expect occasional breaking changes as the spec settles. They also live in their own repository now, open-telemetry/semantic-conventions-genai https://github.com/open-telemetry/semantic-conventions-genai ; the copies still sitting in the main semantic-conventions repo are marked as moved. That split doesn't change any of the attribute names below, but it is the repo to watch. The GenAI semantic conventions in one table the-genai-semantic-conventions-in-one-table The GenAI semantic conventions in one table Almost everything you'll see on a GenAI span https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md comes from this short vocabulary: | What | Attribute / span | Example | |---|---|---| | Operation also the span name prefix | gen ai.operation.name | chat , execute tool , invoke agent , embeddings | | Provider | gen ai.system renamed gen ai.provider.name in newer versions | openai , anthropic , aws.bedrock | | Requested model | gen ai.request.model | gpt-4o-mini | | Tokens in / out | gen ai.usage.input tokens , gen ai.usage.output tokens | 412 / 96 | | Why it stopped | gen ai.response.finish reasons | "stop" , "tool calls" | | Agent | gen ai.agent.name , gen ai.agent.id | travel-concierge | | Tool | gen ai.tool.name , gen ai.tool.type , gen ai.tool.call.id | get weather / function | | Content opt-in | gen ai.input.messages , gen ai.output.messages , gen ai.system instructions | full prompt / response | Span names follow {operation} {model-or-name} , which gives you chat gpt-4o-mini , execute tool get weather , invoke agent travel-concierge . There are also two client metrics https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md : gen ai.client.operation.duration a latency histogram, in seconds and gen ai.client.token.usage a token histogram, split by gen ai.token.type . There is no combined total-tokens attribute. The conventions define input and output only, so if you want a total, sum the two. Step 1: Auto-instrument the OpenAI SDK in Python no code changes step-1-auto-instrument-the-openai-sdk-in-python-no-code-changes Step 1: Auto-instrument the OpenAI SDK in Python no code changes This step is Python, and so is the application code in every step after it. The package names, the launcher and the environment-variable handling below are all specific to the Python instrumentation. If you're on Node or TypeScript, three of those details change, and the last section of this step covers them. The official OpenTelemetry instrumentation for the OpenAI SDK is opentelemetry-instrumentation-openai-v2 https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-openai-v2 , part of opentelemetry-python-contrib and published on PyPI https://pypi.org/project/opentelemetry-instrumentation-openai-v2/ . Install it alongside the OTLP exporter, pinned: pip install opentelemetry-distro==0.65b0 \ opentelemetry-sdk==1.44.0 \ opentelemetry-exporter-otlp-proto-grpc==1.44.0 \ opentelemetry-instrumentation-openai-v2==2.4b0 \ openai==2.50.0 These five were verified working together https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/pyproject.toml on 2026-07-29 under Python 3.14.5. They are pre-1.0 packages, so re-check the set before you pin it into your own build. Something has to be listening on the other end. The endpoint below is http://localhost:4317 , so an OTLP receiver has to be running on that port before any of this produces a visible trace. A Jaeger, Grafana Tempo or Aspire dashboard container covers local debugging; an OpenTelemetry Collector, a vendor's agent or IceGate covers the rest, and Step 6 goes into the choice. Point the exporter at a port with nothing behind it and your app keeps working, because the exporter retries and logs the failure instead of raising. What you get is export warnings on stderr and no traces at the far end, which looks exactly like broken instrumentation when the instrumentation is fine. That is why the recipe's prerequisites list two processes rather than one: Ollama running locally with the model already pulled, and IceGate listening on 4317 for OTLP gRPC. Swap in whichever OTLP backend you already have for the second. Configure export and behavior through environment variables, without writing any code: export OTEL SERVICE NAME=my-llm-app export OTEL EXPORTER OTLP ENDPOINT=http://localhost:4317 export OTEL EXPORTER OTLP PROTOCOL=grpc Read when the instrumentation loads, so it has to be set BEFORE the process starts. Exporting it from Python is too late. Emits the latest GenAI conventions instead of the frozen v1.30 default. export OTEL SEMCONV STABILITY OPT IN=gen ai latest experimental OFF by default. This is an ENUM, not a boolean: NO CONTENT, SPAN ONLY, EVENT ONLY, SPAN AND EVENT. Any other value including true logs a warning and silently falls back to NO CONTENT. See Step 4 to opt in. export OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=NO CONTENT Port, protocol and installed exporter package all have to agree. 4317 is the OTLP/gRPC port and pairs with grpc , which is what the pinned opentelemetry-exporter-otlp-proto-grpc speaks. To export over HTTP instead, move the endpoint to 4318 , set OTEL EXPORTER OTLP PROTOCOL=http/protobuf , and install opentelemetry-exporter-otlp-proto-http https://pypi.org/project/opentelemetry-exporter-otlp-proto-http/ , which is the step people skip. It is a separate distribution https://opentelemetry.io/docs/languages/python/exporters/ , and with only the gRPC package installed the launcher cannot resolve the HTTP exporter and fails at startup. Then run your app under the OpenTelemetry launcher. Every OpenAI call now produces a chat span automatically: opentelemetry-instrument python app.py We verified that on a file importing nothing from OpenTelemetry https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/app zerocode.py . Zero code changes is not zero dependencies, though. The distro and instrumentation packages above still have to be installed before the launcher has anything to discover. If you'd rather wire it up in code, OpenAIInstrumentor .instrument is the call that patches the SDK: python from openai import OpenAI from opentelemetry.instrumentation.openai v2 import OpenAIInstrumentor OpenAIInstrumentor .instrument every OpenAI call now emits a chat span client = OpenAI resp = client.chat.completions.create model="gpt-4o-mini", messages= {"role": "user", "content": "Summarize today's incidents."} , That alone will not export anything. instrument creates spans, but it has no opinion about where they go. Under opentelemetry-instrument the launcher was doing that half of the job, building a TracerProvider with an OTLP exporter out of the environment variables above. Drop the launcher, run plain python app.py , and the global tracer stays the default no-op: the app runs normally and no span ever leaves the process. Nothing in the logs hints at why, because no exporter was ever built to fail at sending one. So replacing the launcher means doing both halves yourself: python from opentelemetry import trace from opentelemetry.exporter.otlp.proto.grpc.trace exporter import OTLPSpanExporter from opentelemetry.instrumentation.openai v2 import OpenAIInstrumentor from opentelemetry.sdk.resources import Resource from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor provider = TracerProvider resource=Resource.create {"service.name": "my-llm-app"} provider.add span processor BatchSpanProcessor OTLPSpanExporter endpoint="http://localhost:4317", insecure=True trace.set tracer provider provider has to happen before instrument OpenAIInstrumentor .instrument ... your app runs here ... provider.shutdown flush before exit The shutdown call at the end fails the same quiet way when you forget it. BatchSpanProcessor queues spans and flushes on a timer, so a short script that does its work and exits takes the pending batch with it, and you are back to an empty backend. Our recipe's telemetry.py https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/telemetry.py handles both halves in one place and returns the provider so the caller can shut it down. One constraint does not move into code at all. OTEL SEMCONV STABILITY OPT IN is read when the instrumentation module loads, so setting os.environ ... above instrument is already too late. You silently get the v1.30 names back gen ai.system instead of gen ai.provider.name and nothing raises to tell you. Export it in your shell or .env before the process starts. That same telemetry.py exits loudly on a missing opt-in instead of quietly emitting the old names. By default the instrumentation records metadata only model, token counts, finish reason, duration and leaves message content out, which is the right default for production. Step 4 covers turning content on safely. Which package to install which-package-to-install Which package to install Use opentelemetry-instrumentation-openai-v2 , the official OTel implementation. The older opentelemetry-instrumentation-openai is Traceloop's community package OpenLLMetry . It's still maintained, but it isn't the standard. For Anthropic, Bedrock, Vertex and others you have a choice: their OpenAI-compatible endpoints, a community or framework instrumentor OpenLLMetry, or framework-native ones for LangChain, LlamaIndex, the OpenAI Agents SDK, CrewAI , or the dedicated agents instrumentation. All of them emit the same gen ai. attributes, so everything below is identical regardless of provider. Pointing at an OpenAI-compatible endpoint pointing-at-an-openai-compatible-endpoint Pointing at an OpenAI-compatible endpoint In practice the SDK requires two things, and the model name has to move with them: python import os from openai import OpenAI client = OpenAI base url="http://localhost:11434/v1", any OpenAI-compatible endpoint api key="ollama", required by the SDK; ignored by the server resp = client.chat.completions.create model=os.environ.get "OLLAMA MODEL", "gemma4:12b-mlx" , messages= {"role": "user", "content": "Summarize today's incidents."} , Change only the base url and you get a model-not-found error instead of a trace. A local Ollama serves exactly the models you have pulled onto that machine, and gpt-4o-mini is not one of them, so the model name has to change along with the endpoint. Our recipe reads it from OLLAMA MODEL , which is why its prerequisites start with ollama pull gemma4:12b-mlx . The emitted provider value describes the SDK and protocol surface, not the backend actually serving the request. Point the OpenAI SDK at Ollama and you get gen ai.provider.name = "openai" , not "ollama" . That matters if you group cost or latency by provider. If you're on Node or TypeScript if-youre-on-node-or-typescript If you're on Node or TypeScript The steps above are Python. The JS path differs in three ways that the rest of this guide doesn't cover: declares @opentelemetry/instrumentation-openai supportedVersions: " =4.19.0 <7" checked against 0.19.0 . With openai 7.x installed it silently never patches: your app runs fine and emits no spans at all. Pin openai to 6.x.- It has no OTEL SEMCONV STABILITY OPT IN support, so its Chat Completions path emits the deprecated gen ai.system . The Step 2 advice below about gen ai.provider.name can't be followed in JS today. - Under Node ESM, calling registerInstrumentations at runtime is not enough. The loader hook has to be registered at process start: node --import tsx --experimental-loader=@opentelemetry/instrumentation/hook.mjs src/index.ts Our TypeScript recipe https://github.com/icegatetech/integrations/blob/main/typescript/openai-ollama/package.json carries both fixes: the openai 6.49.0 pin, and the loader flag in its start script. Step 2: Add agent and tool spans with invoke agent and execute tool step-2-add-agent-and-tool-spans-with-invoke agent-and-execute tool Step 2: Add agent and tool spans with invoke agent and execute tool Auto-instrumentation gives you a flat list of chat spans. What it can't know is your agent's structure: which calls belong to which task, and when a tool ran. You add that by wrapping the run loop in an invoke agent span https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-agent-spans.md and each tool call in an execute tool span. The auto-instrumented chat spans then nest underneath on their own.The sketch below is illustrative pseudocode: call model , dispatch tool and the shape of plan are yours to supply. For a complete runnable version, with a real client call, real tool dispatch and real control flow around the tool-call decision, see agent.py https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/agent.py in the cookbook. The one trap it has to handle comes up after the trace diagram. python from opentelemetry import trace from opentelemetry.instrumentation.openai v2 import OpenAIInstrumentor OpenAIInstrumentor .instrument tracer = trace.get tracer "travel.concierge" AGENT = "travel-concierge" AGENT ID = "travel-concierge-001" def run agent user msg : with tracer.start as current span f"invoke agent {AGENT}", kind=trace.SpanKind.INTERNAL, attributes={ "gen ai.operation.name": "invoke agent", "gen ai.agent.name": AGENT, "gen ai.agent.id": AGENT ID, "gen ai.provider.name": "openai", }, : 1 model decides what to do; auto-instrumented as a child chat span plan = call model user msg 2 wrap each tool call in its own execute tool span for call in plan.tool calls: with tracer.start as current span f"execute tool {call.name}", attributes={ "gen ai.operation.name": "execute tool", "gen ai.tool.name": call.name, "gen ai.tool.type": "function", "gen ai.tool.call.id": call.id, }, : call.result = dispatch tool call.name, call.args 3 model writes the final answer, another child chat span return call model user msg, tool results=plan.tool calls Note gen ai.provider.name rather than gen ai.system . Once you have opted into gen ai latest experimental in Step 1, that is the current name for this attribute; gen ai.system is the name that convention deprecated. Most backends still accept the old spelling through a deprecation fallback, so nothing breaks if you use it, but it contradicts the opt-in you just set. gen ai.agent.id and gen ai.tool.call.id are worth setting too. The provider already returns a tool-call id, and backends with typed columns for these can only populate them if you emit them. The result is a trace with the same shape as the run: With that structure in place you can tell where a 45-second response went: into a slow tool, into a retry loop, or into the model call itself. The trap is tool choice . Set it to "required" on the planning call if you want the tool step to be deterministic, and make sure you don't set it on the final call. Left on, it forces the model to request another tool and the loop never terminates. Step 3: From token usage to LLM cost per request step-3-from-token-usage-to-llm-cost-per-request Step 3: From token usage to LLM cost per request The instrumentation records gen ai.usage.input tokens and gen ai.usage.output tokens on every chat span, so per-request cost is those counts multiplied by your model's price per token. The gen ai.client.token.usage metric lets you spot token-hungry prompts before they reach production, and gen ai.response.finish reasons tells you when a call ended on tool calls rather than stop , which is how you catch runaway tool loops. The token counts land automatically. Currency does not, because that needs a price table, and a fresh deployment may not have one. On IceGate the built-in prices table is filled by a crawler that ships disabled, so cost columns read NULL until you turn it on. Our cost query https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/queries/cost per run.sql is the one a real deployment should run, and on a stock install it returns true token counts next to NULL costs. To get real rates, turn the crawler on with maintain.pricing.enabled in IceGate's Helm values. It ships off because it is the one maintain workload that makes outbound internet calls, and granting the pod egress should be a deliberate choice rather than a default someone inherits. The shipped source list already covers OpenRouter and LiteLLM, so there is usually nothing else to configure, but prices has to exist before the first crawl: the crawler appends to that table and will not create it. Run migrate first. If you only need the arithmetic, skip the table and supply the rates inline. Our worked example https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/queries/cost per run worked example.sql runs the same calculation against a WITH rates AS ... CTE, so it returns real numbers on a stock install with an empty prices table. That is also the query for a local model, which has no upstream rate card at all: the token counts are genuine measurements, and you are pricing them as if a paid model had served them. These chat spans are also big, which matters later. Prompts, completions and tool payloads make a single chat span roughly fifty times the size of an ordinary HTTP or database span, the ~20 KB per span our own pricing model assumes. That size gap is why volume-priced observability gets expensive so fast, and we work the numbers through in our companion piece on why LLM observability bills explode at scale /posts/the-ai-tax-on-telemetry . Step 4: Capture prompts and responses safely without leaking PII step-4-capture-prompts-and-responses-safely-without-leaking-pii Step 4: Capture prompts and responses safely without leaking PII Content capture is off by default, because prompts and responses routinely contain PII. Turn it on with the environment variable from Step 1: export OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=SPAN ONLY The value is an enum, not a boolean: NO CONTENT , SPAN ONLY , EVENT ONLY or SPAN AND EVENT . SPAN ONLY gives you the on-span attributes described below. true and false belong to the older pre-opt-in path and are rejected once gen ai latest experimental is set. The library logs a warning and falls back to NO CONTENT , so you get no content at all and one log line explaining why. With gen ai latest experimental enabled, the instrumentation records content as structured span attributes gen ai.input.messages , gen ai.output.messages , gen ai.system instructions , so you can replay the exact conversation when debugging. In legacy mode it emits the same content as log events instead. In production these attributes are both large and sensitive. Redact at the OpenTelemetry Collector before the data leaves your network, or keep full content only in storage you control. Step 5: Group multi-turn conversations with gen ai.conversation.id step-5-group-multi-turn-conversations-with-gen aiconversationid Step 5: Group multi-turn conversations with gen ai.conversation.id Everything so far covers a single turn. A real agent fields many turns from one user over time, and each turn is its own trace with its own root span. Without a shared key, a backend shows you a pile of unrelated traces that happen to share an agent name, instead of a conversation you can read end to end. The attribute for this is gen ai.conversation.id https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md , which the spec defines as a "unique identifier for a conversation session, thread ". It is a span attribute, not a resource attribute. OpenInference's equivalent is session.id . IceGate populates its typed operations.conversation id column from it, and that column exists to group a conversation's operations across traces.The obvious approach does not work. Setting the id on the invoke agent span you already create in Step 2 leaves it on that one span, because span attributes don't inherit. Every auto-instrumented chat span nested underneath stays empty, which defeats the grouping exactly where it matters: the model calls are the spans you most want to line up across turns. A resource attribute would cover every span for free, and it is still wrong. Resources are process-wide and immutable once the TracerProvider is built; conversations are neither. One process serves many conversations, sequentially or concurrently, and one conversation can span many processes. Bake a conversation id into the resource and it breaks the first time either of those is true. What works is a span processor. Register one whose on start hook onStart in Node reads the active conversation id and stamps it on each span as the span opens. Because on start runs while the span is still live, span.set attribute here is ordinary public API rather than a reach into private SDK internals or into a span that has already ended. Register it as the outermost processor on the TracerProvider and coverage becomes uniform, since every span the process creates passes through the same hook, whether your own code created it or an auto-instrumentation did, deep inside a library call. The active id itself comes from a ContextVar in Python or AsyncLocalStorage in Node. Both are the standard mechanism in their language for ambient per-call context that survives async boundaries. Unlike the sketch in Step 2, this is real working code, taken from conversation.py https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/conversation.py in the cookbook the Node port is : https://github.com/icegatetech/integrations/blob/main/typescript/openai-ollama/src/conversation.ts conversation.ts python from contextlib import contextmanager from contextvars import ContextVar from opentelemetry.sdk.trace import SpanProcessor current conversation id: ContextVar str | None = ContextVar "icegate conversation id", default=None def current id - str | None: """The active conversation id, or None outside any conversation block.""" return current conversation id.get @contextmanager def conversation conversation id: str | None : """Make conversation id the active conversation for the block.""" token = current conversation id.set conversation id try: yield conversation id finally: current conversation id.reset token restores on exception too class ConversationSpanProcessor SpanProcessor : """Stamps gen ai.conversation.id on every span started while a conversation is active.""" def init self, inner: SpanProcessor : self. inner = inner def on start self, span, parent context=None : conversation id = current id reads the ContextVar above if conversation id is not None: span.set attribute "gen ai.conversation.id", conversation id self. inner.on start span, parent context=parent context The wrapper owns the whole SpanProcessor contract, so the remaining hooks have to reach inner . Drop these and batching never flushes. def on end self, span : self. inner.on end span def shutdown self : self. inner.shutdown def force flush self, timeout millis=30000 : return self. inner.force flush timeout millis=timeout millis Wrap the work in conversation "conv-bc3ceb84" and every span opened inside the block carries the id, including the ones auto-instrumentation creates deep inside a library call. Two turns then share one id while staying two separate traces: A real two-turn run, queried straight out of IceGate's operations table, gives 8 spans with two distinct trace id s d1161d75… and e0fd6bbe… and one shared conversation id , conv-bc3ceb84… , on every one of them. Four of those spans are the auto-instrumented chat spans, which the recipe never sets gen ai.conversation.id on directly. The span processor is what reached them, because application code never touches those spans at all. The recipe runs the agent twice https://github.com/icegatetech/integrations/blob/main/python/openai-ollama/recipe.py on purpose. A single turn is a single trace, which would demonstrate nothing about grouping across traces. Step 6: Where to send your gen ai telemetry step-6-where-to-send-your-gen ai-telemetry Step 6: Where to send your gen ai telemetry Any OTLP-compatible backend can receive this telemetry. For local debugging, a Jaeger, Grafana Tempo or Aspire dashboard container works. In production, the vendors that consume the GenAI conventions Datadog, SigNoz, Uptrace and others all accept the same data. That is what instrumenting to the standard buys you. Changing where the data goes is a one-line change instead of a re-instrumentation project. The cost problem from Step 3 gets resolved at the same layer. Point the same OTLP exporter at IceGate, our open-source observability engine, and those heavy chat spans land in open Parquet and Iceberg files on your own object storage, queryable with plain SQL. Because retention sits on cheap object storage rather than a proprietary indexed store, you can keep full-fidelity traces, prompts and tool calls included, instead of sampling them away. Nothing about your instrumentation changes: you keep the open OpenTelemetry data you just emitted, and it stays yours. Our companion piece on the observability data lake /posts/observability-ingestion-on-s3 covers that architecture in depth. IceGate is Apache-2.0 and speaks OTLP natively. Star it on GitHub https://github.com/icegatetech/icegate , point your collector at it, and watch your agent's traces land in open formats. Every step above has a runnable, verified recipe in the integrations cookbook https://github.com/icegatetech/integrations , in both Python and TypeScript. FAQ faq FAQ What are the OpenTelemetry GenAI semantic conventions? what-are-the-opentelemetry-genai-semantic-conventions What are the OpenTelemetry GenAI semantic conventions? A standard set of span names, attributes gen ai. , metrics and events for recording generative-AI operations: the model, tokens, finish reason, agent and tool structure, and optional prompt and response content. Emitting them keeps AI telemetry consistent across frameworks and vendors. How do I instrument an LLM with OpenTelemetry without rewriting my app? how-do-i-instrument-an-llm-with-opentelemetry-without-rewriting-my-app How do I instrument an LLM with OpenTelemetry without rewriting my app? Install the provider's OTel instrumentation for OpenAI, opentelemetry-instrumentation-openai-v2 , set the OTLP environment variables, and run under opentelemetry-instrument . Every model call is traced automatically; add manual invoke agent and execute tool spans only where you want agent and tool structure. Which OTLP port and protocol should I use? which-otlp-port-and-protocol-should-i-use Which OTLP port and protocol should I use? Match all three: 4317 with OTEL EXPORTER OTLP PROTOCOL=grpc and the opentelemetry-exporter-otlp-proto-grpc package, or 4318 with http/protobuf and opentelemetry-exporter-otlp-proto-http . The two exporters are separate distributions, so selecting a protocol whose package isn't installed makes the launcher fail at startup. Which gen ai attributes matter most? which-gen ai-attributes-matter-most Which gen ai attributes matter most? gen ai.request.model , gen ai.usage.input tokens / output tokens for cost , and gen ai.response.finish reasons for loop and error detection . Add gen ai.agent.name and gen ai.tool.name to give traces structure. Which convention version am I emitting? which-convention-version-am-i-emitting Which convention version am I emitting? By default the OpenAI-v2 instrumentation follows conventions around v1.30 and won't record newer details. Set OTEL SEMCONV STABILITY OPT IN=gen ai latest experimental to emit the latest version, including on-attribute content capture. Set it in the environment before the process starts: the instrumentation reads it at import time, so setting it from Python is too late. The provider attribute gen ai.system is also being renamed to gen ai.provider.name in newer versions, so if you opt in, use the new name in your manual spans too. Do I have to capture prompt content? do-i-have-to-capture-prompt-content Do I have to capture prompt content? No. It's off by default, and only metadata is recorded. Enable it deliberately with SPAN ONLY not true , since the setting is an enum , redact at the Collector, and prefer keeping full content in storage you control. Does this work for Anthropic, Bedrock, or my agent framework? does-this-work-for-anthropic,-bedrock,-or-my-agent-framework Does this work for Anthropic, Bedrock, or my agent framework? Yes. Use OpenAI-compatible endpoints, a community or framework instrumentor, or the dedicated agents instrumentation. All of them emit the same gen ai. attributes, so your dashboards and queries don't change when you switch providers. Language is a different matter: see the Node and TypeScript note in Step 1. How do I group a multi-turn conversation into one view? how-do-i-group-a-multi-turn-conversation-into-one-view How do I group a multi-turn conversation into one view? Stamp gen ai.conversation.id on every span, not just the agent span. Span attributes do not inherit, so an id set only on invoke agent leaves every auto-instrumented chat span without it. The pattern that covers all of them is a span processor that reads the active conversation id from a ContextVar Python or AsyncLocalStorage Node in its on start hook, as in Step 5. Trademarks and disclaimer trademarks-and-disclaimer Trademarks and disclaimer Trademarks. OpenTelemetry and Jaeger are trademarks of the Cloud Native Computing Foundation CNCF / The Linux Foundation. Grafana® and Tempo® are trademarks of Raintank, Inc. Grafana Labs . Apache®, Apache Iceberg™, and Apache Parquet™ are trademarks of The Apache Software Foundation. OpenAI® is a trademark of OpenAI OpCo, LLC. Anthropic® is a trademark of Anthropic PBC. Amazon S3®, Amazon Bedrock®, and AWS® are trademarks of Amazon Technologies, Inc. Google® and Vertex AI® are trademarks of Google LLC. Microsoft® is a trademark of Microsoft Corporation. IBM® is a trademark of International Business Machines Corporation. Elasticsearch® is a trademark of Elasticsearch B.V. / Elastic N.V. Datadog® is a trademark of Datadog, Inc. Docker® is a trademark of Docker, Inc. Python® is a trademark of the Python Software Foundation. Node.js® is a trademark of the OpenJS Foundation. SigNoz, Uptrace, Ollama, LangChain, LlamaIndex, CrewAI, Traceloop / OpenLLMetry, OpenLIT, Langtrace, and Scorecard are designations of their respective owners. All other product names, logos, and brands are the property of their respective owners. Names are used solely for identification and do not imply affiliation, sponsorship, or endorsement by the trademark holders. Disclaimer. IceGate and TripleCloud are not affiliated with or endorsed by the Cloud Native Computing Foundation CNCF , The Apache Software Foundation, OpenAI, Anthropic, Amazon Web Services, Google, Microsoft, IBM, Elastic, Grafana Labs, Datadog, or any of the other companies mentioned here. The instructions and version pins above reflect the packages and configurations described, verified on the dates stated, and may not account for newer releases; the OpenTelemetry GenAI semantic conventions are still experimental and subject to breaking change. Corrections and clarifications are welcome, and we will fix any inaccuracies.