You shipped your AI agent. It answers questions, calls tools, retrieves documents, and mostly does the right thing. Mostly. When it does the wrong thing, you have no idea why. The prompt looked fine. The retrieval seemed relevant. Somewhere between the user's question and the final answer, something went sideways, and you are staring at a log that says "200 OK" while your customer stares at nonsense.
APM dashboards show you HTTP latency and error rates. They have no idea what your agent is thinking. An agent is not a request-response cycle. An agent reasons, branches, calls tools, re-evaluates, and sometimes loops. Tracing that execution requires a different kind of observability, one that understands LLM calls, token budgets, tool invocations, and multi-step reasoning flows. OpenTelemetry now provides semantic conventions for exactly this, and Amazon OpenSearch Service gives you the full stack to collect, store, query, and visualize those traces in one place.
OpenTelemetry (OTel) is the open standard for distributed tracing, metrics, and logs. If you run microservices, you probably already emit OTel spans for HTTP calls and database queries. The recent addition of generative AI semantic conventions extends that vocabulary to cover LLM-specific operations. Every model call, every tool invocation, every reasoning step gets a standardized span with attributes like model name, token counts (input and output), temperature, tool names, and agent step identifiers.
A few lines of setup and every agent execution emits a structured trace. A GenAI SDK (Python or TypeScript) provides decorators and auto-instrumentation for popular frameworks: Strands Agents, LangGraph, CrewAI, the OpenAI Agents SDK. It hooks into providers like OpenAI, Anthropic, Amazon Bedrock, and LangChain without requiring you to rewrite application code. You add a few lines of setup, and every agent execution emits a structured trace that captures the full decision tree.
What makes this different from logging model calls yourself? Structure and correlation. Each span carries a trace ID that links it to the parent request. A single user question might trigger three LLM calls, two tool invocations, and a retrieval step. OTel stitches them into a single trace tree with timing, token cost, and dependency relationships intact. Without this correlation, you are left grepping through scattered logs trying to reconstruct what happened.
Structured traces make previously impossible questions answerable without manual debugging.
Where is the agent spending tokens? Token usage maps directly to cost. A trace shows you that 80% of your tokens went to a single retrieval-augmented call that stuffed too much context. Or that the agent retried a tool call three times because the first two responses were malformed. You cannot optimize what you cannot measure, and token-level attribution across a multi-step flow is exactly what agent traces provide.
Why did the agent choose that tool? Agent frameworks make routing decisions: which tool to call, in what order, with what parameters. The trace captures each decision as a span. When the agent calls a calculator instead of a database lookup, the trace shows you the reasoning step and input that led there. That visibility turns "the agent hallucinated" from a mystery into a debuggable event.
How long does each step take? Latency in agentic applications compounds. If each LLM call takes 2 seconds and the agent chains four of them, your user waits 8 seconds before seeing a response. The trace timeline exposes these serial dependencies, letting you identify which calls can be parallelized or cached. The difference between a 2-second and a 10-second agent response is often a single unnecessary sequential step.
Is the agent looping? Agents with autonomous tool use can fall into retry loops or circular reasoning patterns. A trace that shows 15 spans where you expected 4 is an immediate signal. OpenSearch Dashboards renders these as hierarchical trees and directed acyclic graphs (DAGs), making pathological patterns visually obvious before they drain your token budget.
Any OTLP-compatible backend can store these traces. OpenSearch Service stands out because it provides the full pipeline as a managed, unified stack: collection, processing, storage, querying, visualization, alerting, and anomaly detection. No stitching together four separate tools.
From your application to a dashboard, the pipeline has four stages. Your application emits OTel spans through the GenAI SDK. An OpenTelemetry Collector normalizes them using the generative AI semantic conventions. Amazon OpenSearch Ingestion (Data Prepper) routes the OTLP data into your OpenSearch Service domain, creating service maps, correlating traces with logs, and computing RED metrics (Rate, Errors, Duration) automatically. OpenSearch Dashboards then renders agent traces as explorable trees, DAGs, and timelines with token-cost overlays. Open the Agent Traces panel in OpenSearch Dashboards and you see something APM views never show you. It differs from the standard APM view (which monitors HTTP services) in that it understands model calls, token budgets, and reasoning flows as first-class concepts. You see a hierarchical breakdown of each agent execution: which model was called, what it cost, which tools fired, how long each step took, and whether the agent looped or branched. This is not a general-purpose tracing UI adapted for AI. It is a dedicated interface built on OTel's generative AI conventions.
When token cost spikes or an agent loops unexpectedly, OpenSearch Service fires alerts on trace data (fire a Slack notification when token cost per request exceeds a threshold), anomaly detection on agent behavior (flag when a normally 3-step agent suddenly takes 12 steps), and log-to-trace correlation (click from a trace span directly to the error log that explains the failure). These capabilities exist in one domain, under one security model, queried with one language.
Your agent is no longer a black box. OTel's generative AI conventions give you structured, correlated traces across every LLM call, tool invocation, and reasoning step. OpenSearch Service stores those traces, visualizes them as explorable execution graphs, alerts on anomalies, and correlates them with your existing logs and metrics. The operational model for AI agents just caught up with the operational model you already have for microservices.
Think of it as tail -f
for your agent's brain. Except instead of scrolling text, you get a full execution graph with cost attribution, latency breakdowns, and anomaly detection built in. The next time your agent does something inexplicable, you will not be guessing. You will be looking at the trace. To get started, check the OpenSearch Observability platform page and the agent traces documentation. The OpenSearch Observability Stack provides a Docker Compose setup with a preconfigured OTel Collector, Data Prepper, and example applications so you can see agent traces working end to end in minutes.