AI Agent Observability with OpenTelemetry

OpenTelemetry, a CNCF graduated project, provides vendor-neutral tracing, metrics and logs that engineers can use to capture each step of an AI agent's execution path, according to an explainer on AI agent observability. The piece argues that agents chain multiple non-deterministic LLM calls, so a single user request can spawn dozens of internal steps, making silent failures, unpredictable cost and latency, and run-to-run variance hard to diagnose without full tracing of tool selection, context and model outputs.

AI Agent Observability with OpenTelemetry AI agents don't just call a model and return its response. Instead, they orchestrate entire workflows that involve selecting tools for LLMs to use, evaluating guardrails, routing between LLM calls, and managing context across conversation turns. This means that a single user request can spawn dozens of internal steps, each one a potential point of failure, latency, or unexpected cost. AI agent observability is the practice of capturing these execution flows so you can understand what the agent did, why it chose that path, and how it performed. In practice, this means tracing an agent run's execution path, capturing inputs and outputs at each step, and tracking cost, token usage, latency, and error rates. AI agent tracing builds the foundation for AI agent observability. Without tracing, debugging an AI agent becomes a game of trying to guess which event triggered what decision in the agent workflow, and why. Why do AI agents need observability? A request can return successfully while the agent gives an incorrect answer, takes too long on a particular step, or triggers duplicate LLM or tool calls. Agent observability gives engineers the evidence to investigate these problems and compare behavior across runs. Let's look at some of these problems in more detail. LLM non-determinism compounds across agent steps A single LLM call is already non-deterministic: the same prompt can produce different outputs across runs. Agents amplify this because they chain multiple LLM calls together, where the output of one step determines the input of the next. This means variance doesn't just exist at one point. Rather, it compounds across the workflow. A slightly different model response in step one can cause the agent to select a completely different tool in step two, which changes the context passed to step three, and so on. Reproducing a specific agent run becomes nearly impossible without capturing the full execution path. Agents can silently produce incorrect results Traditional software can also return incorrect results without throwing an error. With agents, these failures can be harder to catch because each run can take a different path: the workflow completes without any error, but the agent selected the wrong tool, retrieved stale context, or returned a response that is confidently incorrect. Without observability, these silent failures only surface when a user complains or when downstream systems consume bad output. Tracing each decision in the agentic loop, such as which tool was selected, what context was available, what the model actually returned, is the only reliable way to evaluate whether an agent's "successful" run was actually logically correct. Cost and latency are unpredictable by design Agents decide at runtime how many tools to call and how many LLM roundtrips to make. A simple user query might resolve in one LLM call, while a similar query might trigger several tool calls and model invocations depending on the agent's reasoning. This makes cost and latency inherently unpredictable. Observability lets engineers track token usage, tool call frequency, and per-step latency across runs to identify expensive patterns early. For example, if two models with different pricing tiers perform comparably for a given workload, engineers can switch to the cheaper option — but only if they have the data to make that comparison. How does OpenTelemetry enable AI agent observability? OpenTelemetry https://signoz.io/opentelemetry/ OTel is a CNCF graduated project for instrumenting applications to emit telemetry data primarily traces, metrics, and logs https://signoz.io/blog/three-pillars-of-observability/ in a vendor-neutral format. What makes it uniquely suited for agentic applications is that your visibility extends beyond the AI agent to the entire request: what endpoint did the user call, what DB queries ran, were any background jobs enqueued, did the application make any downstream API calls? Setting up AI agent observability with OpenTelemetry Prerequisites - Python 3.12 or newer. Download the latest version https://www.python.org/downloads/ . - A SigNoz Cloud account https://signoz.io/teams/ for visualizing telemetry data. - An OpenAI API key https://platform.openai.com/api-keys to run the agent. - An API client like Bruno https://www.usebruno.com/ or Postman https://www.postman.com/ to test endpoints. Setting up SigNoz Cloud SigNoz Cloud is an OpenTelemetry-native observability platform that unifies traces, metrics, and logs in a single interface. - Sign up https://signoz.io/teams/ for a free SigNoz Cloud account. - Follow the documentation https://signoz.io/docs/ingestion/signoz-cloud/keys/ to generate an ingestion key. - Keep your workspace region e.g., us , in , eu and ingestion key handy for the application configuration. Running the Demo Application Clone and navigate to the demo application: git clone --filter=blob:none --sparse https://github.com/SigNoz/examples.git git -C examples sparse-checkout set python/otel-ai-agent-observability-demo cd examples/python/otel-ai-agent-observability-demo Create and activate a Python virtual environment: python3.12 -m venv .venv source .venv/bin/activate Install the application and OpenTelemetry Python https://signoz.io/opentelemetry/python/ dependencies: pip install -r requirements.txt The requirements.txt file https://github.com/SigNoz/examples/blob/main/python/otel-ai-agent-observability-demo/requirements.txt pins the following OTel packages: - opentelemetry-distro : Provides a convenient mechanism to configure some of the more common options for Python auto-instrumentation. - opentelemetry-exporter-otlp : Installs the OTLP exporters https://signoz.io/blog/what-is-otlp/ needed to deliver telemetry data to any OpenTelemetry-compatible backend https://signoz.io/blog/opentelemetry-backend/ . - opentelemetry-instrumentation-genai-openai and opentelemetry-instrumentation-genai-openai-agents : Modern OpenAI Agents and OpenAI Python API instrumentation packages from the dedicated OpenTelemetry GenAI repository https://github.com/open-telemetry/opentelemetry-python-genai . Next, run opentelemetry-bootstrap to detect widely used Python libraries like the FastAPI web framework https://signoz.io/blog/opentelemetry-fastapi/ and HTTPX client library and install their corresponding instrumentation packages: opentelemetry-bootstrap --action=install Start the application using opentelemetry-instrument to automatically instrument our application, including the AI agent workflow: OPENAI API KEY="<your-openai-api-key " \ OTEL EXPORTER OTLP ENDPOINT="https://ingest.<your-region .signoz.cloud:443" \ OTEL EXPORTER OTLP HEADERS="signoz-ingestion-key=<your-ingestion-key " \ OTEL SERVICE NAME="otel-ai-agent-observability-demo" \ OTEL RESOURCE ATTRIBUTES="service.version=1.0.0,deployment.environment=dev" \ OTEL EXPORTER OTLP PROTOCOL="http/protobuf" \ OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=span only \ opentelemetry-instrument fastapi run --port 8085 --workers 1 Here is what these OpenTelemetry environment variables https://signoz.io/blog/otel-environment-variables/ configure: - OTEL SERVICE NAME="otel-ai-agent-observability-demo" : Uniquely identifies the application in SigNoz Cloud. - OTEL RESOURCE ATTRIBUTES : Attaches metadata version 1.0.0 , environment dev to all exported telemetry. This is used to organize and filter telemetry in the OpenTelemetry backend. - OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=span only : Captures model prompts, completions, and tool definitions as span attributes on LLM model spans. The FastAPI server starts on port 8085. Calling / returns a quick health check response: .venv ❯ curl "http://127.0.0.1:8085/" {"message":"OpenTelemetry NBA agent demo is running"} The demo is now running. Before exploring the agent code, let’s look at the OpenTelemetry conventions and instrumentation packages used to capture AI agent workflows. Understanding OpenTelemetry’s evolving GenAI conventions While early instrumentation discussions focused solely on "LLM tracing", OpenTelemetry standardizes AI telemetry under the broader umbrella of Generative AI GenAI , using semantic conventions prefixed with gen ai. https://github.com/open-telemetry/semantic-conventions-genai/blob/4f85037ef86e92c510d2ef881a58f1076f6fc0e4/docs/gen-ai/README.md . Because GenAI patterns and agentic harnesses evolve far more rapidly than traditional HTTP or database libraries, OpenTelemetry created a dedicated repository, open-telemetry/opentelemetry-python-genai https://github.com/open-telemetry/opentelemetry-python-genai , to enable maintainers to independently develop and publish these libraries. This migration introduced a cleaner, split-scope architecture : 1. opentelemetry.instrumentation.genai.openai agents : Instruments the agent orchestration harness, emitting spans for workflow execution invoke workflow and agent routing invoke agent . 2. opentelemetry.instrumentation.genai.openai : A companion package that instruments the underlying model calls chat , capturing token usage gen ai.usage.input tokens , gen ai.usage.output tokens , model parameters, and prompt context. The earlier Agents instrumentation https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation-genai/opentelemetry-instrumentation-openai-agents-v2 combined workflow and model-call tracing in a single package. Additionally, because opentelemetry-bootstrap has not yet added these dedicated GenAI packages to its automated discovery list, you must install and track them separately through your requirements.txt or pyproject.toml files. How the NBA agent uses tools, guardrails, and sessions Now that we've seen how OpenTelemetry separates agent orchestration from model calls, let's look at the code behind the NBA agent's workflow. Skip to the next section visualizing-ai-agent-workflows-with-opentelemetry if you want to jump directly to generating and inspecting traces. FastAPI serves our HTTP endpoints, the OpenAI Agents SDK https://openai.github.io/openai-agents-python/ manages the agent loop, and the OpenAI Python API library handles the underlying model calls. This separation is why we instrument both the Agents SDK and the API library: one captures the workflow, while the other captures each model call. The core agentic setup Our web application exposes an NBA Reporter agent that answers questions on NBA topics e.g., general news or finals , performs web research, calculates team win percentages, and retains conversation history across turns via server-managed session persistence. The agent service.py file https://github.com/SigNoz/examples/blob/main/python/otel-ai-agent-observability-demo/app/agent service.py defines the agent configuration and the execution function used by our FastAPI endpoint. We initialize the agent with a system prompt, tools, input guardrails, and model selection. The application uses gpt-6-luna OPENAI MODEL = "gpt-6-luna" in app/config.py , which delivers consistent multi-turn reasoning and tool formatting: NBA AGENT = Agent name="NBA Reporter", instructions=NBA INTERACTIVE PROMPT, tools= WebSearchTool , calculate win percentage , input guardrails= nba content guardrail , model=OPENAI MODEL, The calculate win percentage function tool gives the agent a deterministic way to calculate and display team win percentages without the risk of arithmetic hallucinations: php @function tool def calculate win percentage wins: int, losses: int - str: """Calculates the winning percentage for an NBA team given their wins and losses.""" total games = wins + losses if total games == 0: return ".000" return f"{wins / total games:.3f}".lstrip "0" To prevent misuse or off-topic queries, we define an input guardrail. When a user prompt does not match relevant basketball keywords, the function returns a GuardrailFunctionOutput with tripwire triggered=True : app/guardrails.py python @input guardrail def nba content guardrail context: RunContextWrapper None , agent: Agent, input data: str | list, - GuardrailFunctionOutput: """Ensures the user query is relevant to basketball/NBA.""" keywords = "nba", "basketball", "player", ... truncated for brevity extract the last user message from chat history if isinstance input data, list : latest user message = next item.get "content", "" for item in reversed input data if isinstance item, dict and item.get "role" == "user" , "", input query = latest user message else: input query = input data input query = input query.lower is relevant = any keyword in input query for keyword in keywords if len input query < 5 or is relevant: return GuardrailFunctionOutput tripwire triggered=False, output info=None return GuardrailFunctionOutput tripwire triggered=True, output info={ "reason": "The request is off-topic. Please ask questions relevant to NBA or basketball." }, Setting tripwire triggered=True instructs the Agents SDK to abort execution and raise an InputGuardrailTripwireTriggered exception. We intercept this in a FastAPI exception handler to record the exception event directly on the active OpenTelemetry span before returning an HTTP 400 response: @app.exception handler InputGuardrailTripwireTriggered async def handle guardrail block request: Request, exc: InputGuardrailTripwireTriggered, - JSONResponse: guardrail msg = exc.guardrail result.output.output info span = trace.get current span span.record exception exc return JSONResponse status code=400, content={"detail": guardrail msg} Managing session context across turns The run agent turn function completes our implementation. The agent uses the OpenAIConversationsSession object to persist and gather conversation context across turns. If no session id was passed, the session object triggers the creation of a new conversation session. In a follow-up request, the user can attach this ID to ensure the agent has context of the conversation thus far. app/agent service.py python def run agent turn topic: str, user message: str | None, session id: str | None, - dict: nba topic = validate topic topic prompt = build nba turn prompt nba topic, user message if no session ID was given, the sdk internally creates a session id during the turn subsequent calls which pass the ID maintain the conversation context session = OpenAIConversationsSession conversation id=session id result = Runner.run sync NBA AGENT, prompt, session=session message = sanitize agent message result.final output or "" .strip ... truncated for brevity return { "topic": topic, "session id": session.session id, "message": message, "model": OPENAI MODEL, "usage": usage, } Visualizing AI agent workflows with OpenTelemetry Let’s send the agent a few requests to generate traces, then inspect how tool calls, conversation history, and guardrail failures appear in SigNoz Cloud. Interacting with the agent Let’s start by asking the NBA Reporter agent a question about the Lakers' win percentage in the 2025-26 season, as well as LeBron and his future with the team. This query asks three distinct questions, setting us up for follow-up questions in the next turn, and the win percentage ask should trigger the calculate win percentage tool. The agent responds first with the win percentage calculation — the format matches our tool's response shape. It makes a prediction about the upcoming NBA season, and informs us that LeBron is no longer with the LA Lakers. The response JSON also contains the model name, token usage details at a glance, and the session id to continue the conversation in follow-up requests. Let's now add the session id to our request payload and ask why LeBron switched teams. We'll refer to LeBron as he to test the agent's context retrieval capabilities. The agent correctly identifies he as LeBron in the context of the conversation, and outlines why he moved to a new team. It also returns the same session id in the response, further confirming continuation of the previous conversation. Finally, let’s ask an off-topic question to see whether the input guardrails trigger to prevent users from abusing our agent: Because the prompt lacks basketball-related keywords, the tripwire fires immediately. This helps the agent abort execution before executing tools or calling the model, preventing unneeded API spend and latency. Exploring the trace hierarchy in SigNoz Cloud Now let's visualize in SigNoz Cloud the multi-step trace that invoked tool use: Modern OpenTelemetry GenAI instrumentation structures the request into a clean, hierarchical span tree: POST /agent/turn fastapi Server 26.00s ├─ POST /agent/turn http receive asgi Internal ├─ POST /v1/conversations httpx Client ├─ GET /v1/conversations/{id}/items?order=asc httpx Client ├─ invoke workflow Agent workflow genai.openai agents Internal 24.41s │ ├─ POST /v1/conversations/{id}/items httpx Client │ └─ invoke agent NBA Reporter genai.openai agents Internal 23.71s │ ├─ chat gpt-6-luna genai.openai Client 3.94s │ │ └─ POST /v1/responses httpx Client 3.84s │ ├─ execute tool calculate win percentage genai.openai agents Internal │ ├─ execute tool calculate win percentage genai.openai agents Internal │ ├─ POST /v1/conversations/{id}/items httpx Client │ ├─ chat gpt-6-luna genai.openai Client 18.21s │ │ └─ POST /v1/responses httpx Client 17.90s │ └─ POST /v1/conversations/{id}/items httpx Client ├─ POST /v1/traces/ingest ×3 httpx Client └─ POST /agent/turn http send ×2 asgi Internal This trace highlights several key structural details of the agentic loop: 1. Full-stack request lifecycle : The trace begins at the FastAPI boundary POST /agent/turn . Before the agent workflow executes, child HTTP spans show the session layer fetching conversation history GET /v1/conversations/{id}/items . 2. Dedicated agentic scopes : The agent orchestration harness is captured under opentelemetry.instrumentation.genai.openai agents via the invoke workflow and invoke agent spans, defining the agent's decision boundary. 3. Iterative tool execution loop : Rather than a single model completion, the trace maps the full multi-step loop: - Initial model call chat gpt-6-luna , 3.94s : The model processes the turn, detects the win percentage query, and emits tool call instructions. - Tool execution execute tool calculate win percentage ×2 : The agent loop executes our tools, followed by an HTTP span persisting the tool results to the conversation session. - Final synthesis call chat gpt-6-luna , 18.21s : The model receives the tool outputs and generates the comprehensive user-facing response. 4. Standardized GenAI attributes : Selecting either chat gpt-6-luna span reveals attributes for: - Provider and model : gen ai.provider.name , gen ai.request.model , and gen ai.response.model . - Operation and outcome : gen ai.operation.name and gen ai.response.finish reasons . - Token usage : gen ai.usage.input tokens and gen ai.usage.output tokens . Inspecting message content and privacy controls Because we set OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=span only , the instrumentation populates gen ai.input.messages , gen ai.output.messages , gen ai.system instructions , and gen ai.tool.definitions directly as span attributes. This allows engineers to inspect the exact prompts, system instructions, and model completions for each turn during debugging. In environments with sensitive data or strict PII compliance, prompt content capture can be turned off entirely by setting: export OTEL INSTRUMENTATION GENAI CAPTURE MESSAGE CONTENT=no content This retains model timing, token counts, and span hierarchies while omitting raw message payloads and input/output values from the exported telemetry. Handling guardrail failures in traces When an off-topic query triggers the input guardrail tripwire, the agent stops execution before calling the model. In our FastAPI exception handler, intercepting InputGuardrailTripwireTriggered and invoking span.record exception exc captures the exception event on the root span. Monitoring agent performance and metrics While individual traces let you debug specific agent executions, understanding production behavior requires aggregated metrics across turns, models, and sessions. The OpenTelemetry GenAI instrumentation automatically emits standard metrics—such as gen ai.client.operation.duration and token usage counters—alongside your traces. SigNoz Cloud captures this telemetry in a dedicated AI Observability interface that aggregates operational health, latency percentiles, and agent loop behaviors in one unified view. Executive KPIs and contextual drilldowns The top-level overview surfaces high-level operational indicators across your agent infrastructure: - Total cost and token volume : Tracks spend and token consumption across services to help identify expensive workloads. - Latency percentiles and TTFT : Tracks tail latency and Time to First Token TTFT for streaming responses. - LLM call volume and error rates : Monitors traffic spikes and model failure rates across services. Use the dashboard's trace and log drilldowns to track the corresponding telemetry responsible for the numbers. Break down individual metrics by attributes such as model name or service, then combine those breakdowns with dashboard-level filters to answer questions like: - Which model cost you the most for a particular service in the given time frame? - How does LLM latency compare between staging and production over time? - Could differences between your team's test requests and customer requests explain the gap? Evaluating model latency and response times Multi-turn reasoning and tool selection introduce variable latency across agent interactions. The LLM latency by model table and time-series charts track execution performance across percentiles: - Percentile breakdowns : Compare p50, p90, and p95 latencies by model to see whether slow responses affect typical requests or only the tail. - Interactive time-series : Drill into traces around a latency spike to inspect how much time was spent in model calls and tool execution. Diagnosing agent loops with finish reasons The Finish reasons panel shows how model invocations end: - "stop" : Indicates the model finished generating a final response to the user. - "tool calls" : Indicates the model requested one or more tool executions, prompting the agent loop to execute the tool and start another turn. Spikes in "tool calls" relative to "stop" can be signals of recursive agent behavior, repetitive tool retries, or agents struggling to synthesize an answer. Tool call performance and span distribution The Tool calls section aggregates telemetry from the tools invoked by the agent: - Tool call rate and error tracking : Shows how often each tool runs and fails, helping identify repeated calls or unreliable tools. - Tool duration : Shows which tools contribute the most latency to an agent run. - Top span names : Compares agent, model, and tool span counts. Unexpected differences, such as a large gap between agent spans and LLM spans, can guide further trace inspection. Conclusion As AI applications evolve from simple model prompts into autonomous, multi-turn agents, traditional debugging approaches fall short. Agent workflows are non-deterministic, span multiple tool calls, and rely on dynamic context that can silently degrade in quality. Instrumenting your agents with OpenTelemetry delivers end-to-end visibility across every layer: - Full request context : Correlate incoming HTTP requests, session storage lookups, agent decision loops, and downstream API calls in a single trace. - Vendor-neutral GenAI standards : Leverage official OpenTelemetry semantic conventions gen ai. to standardize metrics, attributes, and span structures without proprietary lock-in. - Unified debugging in SigNoz Cloud : Combine traces, metrics, and logs in one interface to diagnose hallucinated tool calls, identify latency bottlenecks, and govern token spend across production runs. If you’re building agentic applications, sign up for a free SigNoz Cloud trial https://signoz.io/teams/ to start observing your AI agents with OpenTelemetry today.