{"slug": "observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production", "title": "Observability for AI Agents: Tracing, Evaluation and Cost Control in Production", "summary": "A developer outlined an approach to instrumenting agentic AI systems with OpenTelemetry, using traces, metrics and logs to follow multi-step agent workflows such as LLM calls, tool invocations and subagent delegation. The writeup notes that as of November 2026 the OpenTelemetry JavaScript documentation lists traces and metrics as Stable while logs remain in Development, and that GenAI semantic conventions should be assessed separately from the maturity of the underlying signals.", "body_md": "*How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices.*\n\nTraditional backend observability is built around relatively predictable execution paths.\n\nA request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred.\n\nAI agents introduce a different level of complexity.\n\nAn agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input.\n\nConsider a relatively simple task:\n\nA user asks an AI assistant to analyze a merge request and identify potential problems.\n\nBehind the scenes, the application might:\n\nThe user sees one response. The application has actually executed a multi-step workflow involving several components.\n\nNow imagine the response takes 45 seconds instead of the expected 8 seconds.\n\nWhere should we investigate?\n\nTraditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why.\n\n**AI observability is about understanding the entire execution path, not just measuring the final API response.**\n\nThis is where OpenTelemetry becomes particularly useful.\n\n** is an open-source observability framework for collecting, processing, and exporting telemetry data.\n\nIt provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms.\n\nInstead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported.\n\nOpenTelemetry is built around three primary signals:\n\n| Signal | What it tells us | AI agent example | \n|---|---|---|\n| Traces | How an operation was executed | Model call → tool → subagent | \n| Metrics | How the system performs over time | Token usage, latency, error rate | \n| Logs | What happened at a specific point | Tool timeout or agent exception | \n\nThese signals complement each other.\n\nA metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation.\n\nOpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector.\n\nAn important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal.\n\n**References:** [OpenTelemetry Documentation](https://opentelemetry.io/docs/) · [OpenTelemetry JavaScript](https://opentelemetry.io/docs/languages/js/)\n\nBefore discussing AI-specific instrumentation, we need to understand traces and spans.\n\nA **trace** represents the complete execution path of an operation.\n\nA **span** represents one individual unit of work within that execution.\n\nFor example:\n\n```\nTrace: POST /api/agent\n|\n+-- HTTP Request\n    |\n    +-- Coordinator Agent\n        |\n        +-- LLM Generation\n        |\n        +-- Tool: GitLab - Get Merge Request\n        |\n        +-- Tool: GitLab - Get Changed Files\n        |\n        +-- Security Agent\n        |   |\n        |   +-- LLM Generation\n        |   |\n        |   +-- Tool: Security Analysis\n        |\n        +-- Testing Agent\n        |   |\n        |   +-- LLM Generation\n        |   |\n        |   +-- Tool: Test Coverage\n        |\n        +-- Final Response Generation\n```\n\nEach span has information such as:\n\nThis creates a hierarchical representation of the complete execution.\n\nThe key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them.\n\nFor AI agents, this distinction is important because one user request can generate many independent operations.\n\nIn distributed systems, a trace often crosses multiple services.\n\nImagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service.\n\nWithout context propagation, every service may generate its own independent trace.\n\nWe would see several unrelated operations instead of one complete workflow.\n\nOpenTelemetry uses W3C Trace Context as its standard propagation format. The `traceparent` header carries trace and parent-span information between instrumented services.\n\nThe same principle applies to agent-based architectures.\n\nWhen an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution.\n\nAutomatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction.\n\nThe important point is that **protocol-level interoperability and observability context propagation are separate concerns**. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace.\n\n**References:** [OpenTelemetry Traces](https://opentelemetry.io/docs/concepts/signals/traces/) · [Context Propagation](https://opentelemetry.io/docs/concepts/context-propagation/) · [W3C Trace Context](https://www.w3.org/TR/trace-context/)\n\nOne of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers.\n\nFor example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes.\n\nThis makes it difficult to compare different implementations using the same monitoring platform.\n\nOpenTelemetry addresses this through **Semantic Conventions**.\n\nSemantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently.\n\nThe GenAI semantic conventions extend this approach to generative AI applications.\n\nSome important attributes include:\n\n| Attribute | Purpose | \n|---|---|\n| `gen_ai.provider.name` | Identifies the model provider | \n| `gen_ai.request.model` | Identifies the requested model | \n| `gen_ai.response.model` | Identifies the actual response model | \n| `gen_ai.operation.name` | Describes the operation | \n| `gen_ai.usage.input_tokens` | Input token usage | \n| `gen_ai.usage.output_tokens` | Output token usage | \n| `gen_ai.conversation.id` | Associates operations with a conversation | \n| `gen_ai.agent.name` | Identifies the agent | \n| `gen_ai.tool.name` | Identifies the executed tool | \n\nFor example, an LLM invocation could be represented by:\n\n```\n{\n  \"gen_ai.operation.name\": \"chat\",\n  \"gen_ai.provider.name\": \"openai\",\n  \"gen_ai.request.model\": \"example-model\",\n  \"gen_ai.usage.input_tokens\": 1200,\n  \"gen_ai.usage.output_tokens\": 350\n}\n```\n\nThis is a simplified span attribute example.\n\nThe benefit is that observability platforms do not need to understand a different custom schema for every application.\n\nThe GenAI semantic conventions also define operations for agents and workflows.\n\nSome examples include:\n\n`invoke_agent`` invoke_workflow``execute_tool`` plan``retrieval`\nThis is particularly valuable for multi-agent architectures.\n\nWe can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents.\n\n```\ninvoke_workflow merge-request-review\n|\n+-- invoke_agent coordinator\n|   |\n|   +-- chat model\n|   +-- execute_tool fetch_merge_request\n|\n+-- invoke_agent security-agent\n|   |\n|   +-- chat model\n|   +-- execute_tool analyze_vulnerabilities\n|\n+-- invoke_agent testing-agent\n|   |\n|   +-- chat model\n|   +-- execute_tool analyze_test_coverage\n|\n+-- Final Aggregation\n```\n\nThis makes the execution model much easier to understand.\n\n**An important maturity consideration:** As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract.\n\nThe current convention documents include:\n\nSince many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration.\n\nThe AI SDK already supports OpenTelemetry-based instrumentation.\n\nIt can generate spans for:\n\nThis means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility.\n\nFor an AI SDK v7 application:\n\n```\npnpm add @ai-sdk/otel @vercel/otel\n```\n\nThe OpenTelemetry integration is provided through `@ai-sdk/otel`.\n\nThis is an important version-specific detail.\n\nIn AI SDK v7, OpenTelemetry integration moved out of the core `ai` package. Older examples using the v6 `experimental_telemetry` configuration should not be copied directly into a v7 application.\n\nCreate an `instrumentation.ts` file in the project root.\n\n``` js\nimport { registerOTel } from '@vercel/otel';\nimport { registerTelemetry } from 'ai';\nimport { OpenTelemetry } from '@ai-sdk/otel';\n\nexport function register() {\n  registerOTel({\n    serviceName: 'ai-agent-platform',\n  });\n\n  registerTelemetry(\n    new OpenTelemetry({\n      recordInputs: false,\n      recordOutputs: false,\n    }),\n  );\n}\n```\n\nThere are two responsibilities here:\n\n`registerOTel` initializes the OpenTelemetry infrastructure for the Next.js application.`registerTelemetry` registers the AI SDK-specific integration that creates GenAI spans.\nI have disabled input and output capture in this example intentionally.\n\nAI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision.\n\nWe will return to this in the security section.\n\nFor a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup.\n\nOnce telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration.\n\nWe can also provide a `functionId` to identify the business operation or agent.\n\n``` js\nimport { generateText } from 'ai';\n\nconst result = await generateText({\n  model: myModel,\n\n  instructions: `\n    You are a financial analysis assistant.\n    Analyze the supplied information carefully.\n  `,\n\n  prompt: 'Analyze the latest portfolio performance.',\n\n  telemetry: {\n    functionId: 'portfolio-analysis-agent',\n  },\n});\n```\n\nThe `functionId` helps distinguish operations in traces.\n\nFor example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as:\n\n```\ninvoke_agent portfolio-analysis-agent\n```\n\nThe AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions.\n\nFor `generateText` and `streamText`, the integration records three main types of spans:\n\n**1. Agent invocation span**\n\nThe root span represents the complete generation operation, including its execution steps and tool calls.\n\nExample:\n\n```\ninvoke_agent portfolio-analysis-agent\n```\n\n**2. Step and model-call spans**\n\nEach model interaction is represented by step and provider-call spans.\n\nThese allow us to understand how many model calls occurred during an operation and how long each one took.\n\n**3. Tool execution spans**\n\nEach client-side tool execution receives its own span.\n\n```\ninvoke_agent portfolio-analysis-agent\n|\n+-- step 0\n|   |\n|   +-- chat example-model\n|   +-- execute_tool get_portfolio\n|\n+-- step 1\n|   |\n|   +-- chat example-model\n|   +-- execute_tool get_market_data\n|\n+-- step 2\n    |\n    +-- chat example-model\n```\n\nThe resulting trace provides a much more complete representation of an agent workflow than a single model request log.\n\nThe integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available.\n\nSometimes the standard GenAI attributes are not enough.\n\nFor example, we may want to know which business workflow initiated an operation or which application component owns it.\n\nThe AI SDK provides `enrichSpan` for adding custom attributes:\n\n``` js\nimport { registerTelemetry } from 'ai';\nimport { OpenTelemetry } from '@ai-sdk/otel';\n\nregisterTelemetry(\n  new OpenTelemetry({\n    recordInputs: false,\n    recordOutputs: false,\n\n    enrichSpan: ({ spanType, operationId }) => ({\n      'app.component': 'portfolio-assistant',\n      'app.operation_id': operationId,\n      'app.span_type': spanType,\n    }),\n  }),\n);\n```\n\nThe exact context available to `enrichSpan` depends on the span and the SDK's integration API.\n\nThe objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry.\n\n**References:** [AI SDK Telemetry](https://ai-sdk.dev/docs/ai-sdk-core/telemetry) · [Next.js OpenTelemetry](https://nextjs.org/docs/app/guides/open-telemetry) · [AI SDK v7 Migration Guide](https://github.com/vercel/ai/blob/main/skills/migrate-ai-sdk-v6-to-v7/SKILL.md)\n\nBuilt-in instrumentation is useful, but it does not automatically understand every application-level decision.\n\nFor example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval.\n\nThese are application-level operations.\n\nWe should instrument them explicitly.\n\nOpenTelemetry provides a tracing API for creating custom spans.\n\nConsider a custom agent orchestration function:\n\n``` js\nimport {\n  trace,\n  SpanStatusCode,\n} from '@opentelemetry/api';\n\nconst tracer = trace.getTracer('agent-orchestrator');\n\nexport async function runReviewWorkflow() {\n  return tracer.startActiveSpan(\n    'workflow.merge-request-review',\n    async (span) => {\n      span.setAttribute(\n        'workflow.name',\n        'merge-request-review',\n      );\n\n      span.setAttribute(\n        'workflow.version',\n        '1.0.0',\n      );\n\n      try {\n        const result = await executeReview();\n\n        span.setStatus({\n          code: SpanStatusCode.OK,\n        });\n\n        return result;\n      } catch (cause) {\n        const error =\n          cause instanceof Error\n            ? cause\n            : new Error(String(cause));\n\n        span.recordException(error);\n\n        span.setStatus({\n          code: SpanStatusCode.ERROR,\n        });\n\n        throw error;\n      } finally {\n        span.end();\n      }\n    },\n  );\n}\n```\n\nAny supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated.\n\nWe now have two levels of observability:\n\nThis distinction is important.\n\nWe should not expect a model instrumentation library to understand the complete business meaning of an application request.\n\nFor A2A-based systems, distributed tracing becomes even more important.\n\nImagine a coordinator running in one service and a security agent running in another service.\n\nThe coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated.\n\nConceptually:\n\n```\nTrace: merge-request-review\n|\n+-- Coordinator Agent\n|   |\n|   +-- Model Call\n|   |\n|   +-- A2A: Delegate to Security Agent\n|       |\n|       +-- Security Agent\n|           |\n|           +-- Model Call\n|           |\n|           +-- Tool Execution\n|\n+-- Final Aggregation\n```\n\nThe HTTP request must carry the appropriate trace context, typically through W3C `traceparent` and `tracestate` headers.\n\nFor custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context.\n\nThis is how we avoid fragmented traces across independently deployed agents.\n\nCollecting telemetry is not enough.\n\nWe need to define which metrics are useful for understanding the health and quality of an agent platform.\n\nI would divide AI observability metrics into five categories.\n\nThese describe how long operations take.\n\n| Metric | Why it matters | \n|---|---|\n| End-to-end workflow duration | Measures total user-facing execution time | \n| Model request duration | Identifies slow inference calls | \n| Time to first token | Measures perceived response latency | \n| Tool execution duration | Identifies slow external operations | \n| Agent invocation duration | Measures the execution time of one agent | \n| Queue wait duration | Identifies scheduling bottlenecks | \n\nFor streaming applications, time to first token is particularly useful.\n\nA workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds.\n\nThese help us understand whether workflows complete successfully.\n\nExamples:\n\nA high retry rate can be an early indicator of a problematic integration or an unstable external dependency.\n\nWe should also distinguish between a successful model response and a successful business workflow.\n\nAn LLM can return a perfectly valid response while the overall task fails to achieve its objective.\n\nToken usage should be tracked at multiple levels.\n\n| Metric | Description | \n|---|---|\n| Input tokens | Tokens sent to the model | \n| Output tokens | Tokens generated by the model | \n| Cached input tokens | Tokens served from provider cache | \n| Reasoning tokens | Model-reported reasoning token usage | \n| Tokens per workflow | Total usage across an entire task | \n| Tokens per agent | Usage attributed to an individual agent | \n\nFor multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response.\n\nThese metrics describe how efficiently agents perform their work.\n\nAn agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem.\n\nThe trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs.\n\nThis is where AI observability starts to overlap with AI evaluation.\n\nThese metrics cannot always be derived from execution telemetry alone.\n\nWe need an evaluation strategy to measure the actual quality of the result.\n\nOne of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow.\n\nAn agent-based application may consume resources across several layers:\n\nThe total cost should reflect the entire execution path.\n\nA useful starting point is to define a model cost formula:\n\n[\n\nC_{\\text{model}} =\n\n\\sum_{i=1}^{n}\n\n\\left(\n\nT_{\\text{input},i}P_{\\text{input},i}\n\n+\n\nT_{\\text{output},i}P_{\\text{output},i}\n\n\\right)\n\n]\n\nWhere:\n\nThe pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities.\n\nWe should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets.\n\nThe complete workflow cost can then be represented as:\n\n[\n\nC_{\\text{workflow}} =\n\nC_{\\text{model}}\n\n+\n\nC_{\\text{tools}}\n\n+\n\nC_{\\text{compute}}\n\n+\n\nC_{\\text{storage}}\n\n]\n\nThis may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available.\n\nImagine two implementations:\n\n| Metric | Agent A | Agent B | \n|---|---|---|\n| Average cost per attempt | $0.04 | $0.07 | \n| Successful completion rate | 90% | 98% | \n| Cost per successful task | ~$0.044 | ~$0.071 | \n\nAgent A is cheaper per attempt, but Agent B completes more tasks successfully.\n\nNeither is automatically better.\n\nThe correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts.\n\nA useful metric is:\n\n[\n\n\\text{Cost per successful task} =\n\n\\frac{\\text{Total execution cost}}\n\n{\\text{Number of successful tasks}}\n\n]\n\nThe numerator should include failed attempts that consumed resources, rather than counting only successful runs.\n\nObservability data can also help us make better model-routing decisions.\n\nThe important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request.\n\nThe current GenAI token metric conventions include proposed counters such as:\n\n`gen_ai.client.inference.usage.input_tokens`` gen_ai.client.inference.usage.output_tokens``gen_ai.client.inference.usage.cache_read.input_tokens`` gen_ai.client.inference.usage.reasoning.output_tokens`\nThese conventions are still in Development.\n\nThere is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics.\n\nA span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations.\n\nFor production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices.\n\nThis makes historical cost calculations more reliable when provider pricing changes.\n\n**References:** [GenAI Token Metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-token-metrics.md) · [GenAI Metrics](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md)\n\nThis distinction is worth emphasizing.\n\nObservability tells us what happened during execution.\n\nEvaluation tells us whether the result was good enough.\n\nA trace can show that an agent:\n\nBut none of these facts proves that the answer was correct.\n\nConsider a RAG application.\n\nThe retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information.\n\nFrom an operational perspective, the request succeeded.\n\nFrom a quality perspective, it failed.\n\nWe need both observability and evaluation.\n\nOffline evaluation measures an AI application against a predefined dataset.\n\nFor example, we can create 100 representative questions and expected outcomes.\n\nWhenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again.\n\nThis helps identify regressions before deploying changes.\n\nUseful evaluation dimensions include:\n\nOnline evaluation measures actual production interactions.\n\nWe can select a sample of production traces and evaluate them using:\n\nThe important part is to turn real production failures into future test cases.\n\nThis creates a continuous improvement loop.\n\n```\nProduction Traces\n       |\n       v\nFailure Analysis\n       |\n       v\nEvaluation Dataset\n       |\n       v\nOffline Evaluation\n       |\n       v\nPrompt / Model / Agent Changes\n       |\n       v\nDeployment\n       |\n       v\nOnline Evaluation\n       |\n       +------> New Regression Cases\n```\n\nTwo examples of AI-focused observability and evaluation platforms are:\n\n**Langfuse**\n\nProvides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements.\n\nDocumentation: [https://langfuse.com/docs/evaluation/overview](https://langfuse.com/docs/evaluation/overview)\n\n**Arize Phoenix**\n\nAn open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions.\n\nIt provides tracing, evaluation, datasets, experiments, and debugging capabilities.\n\nDocumentation: [https://phoenix.arize.com/](https://phoenix.arize.com/)\n\nThese tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy.\n\n**Reference:** [Langfuse Evaluation Concepts](https://langfuse.com/docs/evaluation/core-concepts) · [Arize Phoenix](https://github.com/Arize-ai/phoenix)\n\nIn a production architecture, applications should not necessarily export telemetry directly to every observability backend.\n\nThis is where the OpenTelemetry Collector becomes useful.\n\nThe Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations.\n\nIt provides a central place for:\n\nA simplified architecture looks like this:\n\n```\n                 AI Applications\n                       |\n            +----------+----------+\n            |          |          |\n            v          v          v\n         Next.js     .NET API   Python Agent\n            |          |          |\n            +----------+----------+\n                       |\n                      OTLP\n                       |\n                       v\n             OpenTelemetry Collector\n                       |\n              +--------+--------+\n              |        |        |\n              v        v        v\n            Traces   Metrics   Logs\n              |        |        |\n              v        v        v\n            Jaeger   Prometheus  Log Backend\n              |\n              v\n            Grafana\n```\n\nThe same telemetry can also be routed to AI-specific observability platforms where supported.\n\nIn enterprise environments, we may have several applications and teams using different frameworks and languages.\n\nA Collector provides a common ingestion point and makes it possible to apply centralized policies.\n\nFor example, a Collector can filter sensitive attributes before telemetry reaches the final backend.\n\nIt also allows us to change exporters or observability providers without changing the instrumentation in every application.\n\nOpenTelemetry supports different Collector deployment patterns.\n\nA common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors.\n\nThis provides a separation between local telemetry collection and centralized processing.\n\nHowever, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling.\n\n**Reference:** [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) · [Gateway Deployment Pattern](https://opentelemetry.io/docs/collector/deploy/gateway/)\n\nAgent-based workflows can generate a significant amount of telemetry.\n\nConsider an agent that performs 15 model calls and 20 tool calls during a single task.\n\nIf the application handles thousands of tasks per hour, storing every span indefinitely may become expensive.\n\nSampling helps control this volume.\n\nThere are two common approaches.\n\nThe sampling decision is made when the trace starts.\n\nFor example, we may decide to sample 10% of incoming requests.\n\nThis is relatively simple and reduces the amount of telemetry generated or exported.\n\nHowever, we do not yet know whether a trace will eventually fail or become unusually slow.\n\nThe sampling decision is made after considering the trace's collected spans.\n\nThis makes it possible to retain:\n\nFor AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed.\n\nHowever, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance.\n\nThis generally requires trace-ID-aware routing when using multiple Collector instances.\n\nA practical approach is to combine sampling strategies:\n\n**Reference:** [OpenTelemetry Sampling](https://opentelemetry.io/docs/concepts/sampling/) · [Tail Sampling Processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor)\n\nThis is one of the most important aspects of AI observability in enterprise environments.\n\nTraditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names.\n\nAI telemetry can contain much more sensitive information.\n\nSending all this information to an observability backend can create additional privacy and data governance risks.\n\nA particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise.\n\nWe can disable this behavior:\n\n```\nregisterTelemetry(\n  new OpenTelemetry({\n    recordInputs: false,\n    recordOutputs: false,\n  }),\n);\n```\n\nFor many production applications, this is a better starting point.\n\nWe can still collect useful operational telemetry:\n\nwithout exporting the complete conversation.\n\nBefore exporting telemetry, ask whether each attribute is genuinely needed.\n\nAvoid sending:\n\nFor sensitive workflows, use approved metadata rather than full payloads.\n\nAn observability backend should not automatically become another storage system for business data.\n\nThe fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely.\n\nEnterprise environments should define:\n\nA sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production.\n\n**Reference:** [OpenTelemetry GenAI Content Capture](https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-spans.md)\n\nOpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself.\n\nNot every framework, provider, or agent runtime exposes the same information.\n\nSome integrations can automatically trace model calls and tool execution, while others require custom instrumentation.\n\nWe should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility.\n\nThe GenAI conventions are still under development.\n\nAttribute names, operation models, and implementation support may evolve.\n\nTeams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity.\n\nA trace can explain what happened but does not prove that the model's result is correct.\n\nEvaluation datasets, deterministic checks, and human feedback remain necessary.\n\nCollecting every input, output, tool argument, and execution event can generate significant storage and processing overhead.\n\nWe should measure the cost of observability itself.\n\nAdding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series.\n\nUse traces for request-level correlation and keep metric dimensions bounded.\n\nA2A communication, queues, background tasks, and remote tool execution may cross process boundaries.\n\nIf context is lost, we can end up with disconnected traces.\n\nLLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased.\n\nA single evaluation score should not become the only definition of quality.\n\nCombine deterministic checks, representative datasets, human review, and calibrated evaluators.\n\nI think this is where observability moves from an implementation detail to an engineering capability.\n\nWhen multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms.\n\nOver time, this creates fragmented visibility.\n\nOne team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model.\n\nAs an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications.\n\nEstablish a baseline for the telemetry every agent-based application should produce.\n\n| Category | Required information | \n|---|---|\n| Application | Service name, environment, version | \n| Workflow | Workflow name, workflow version | \n| Agent | Agent name, execution status | \n| Model | Provider, requested model, response model | \n| Usage | Input/output tokens, cache usage | \n| Tools | Tool name, duration, execution status | \n| Execution | Trace ID, span ID, error information | \n| Business outcome | Completed, failed, canceled, escalated | \n\nThe objective is to make observability consistent without forcing every team to use the same agent framework.\n\nWe can create an internal TypeScript package that wraps the common OpenTelemetry configuration.\n\n```\n@company/ai-observability\n|\n+-- OpenTelemetry initialization\n+-- AI SDK integration\n+-- Agent span utilities\n+-- Workflow instrumentation\n+-- Tool execution tracking\n+-- Cost attribution\n+-- Evaluation metadata\n+-- Privacy and redaction policies\n```\n\nThis package can provide standard helpers for application-level tracing and reduce duplicated implementation.\n\nFor teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations.\n\nWe should define a common set of metrics that helps teams understand both technical performance and business outcomes.\n\nA shared dashboard can make it easier to compare systems and identify performance regressions.\n\nObservability without evaluation gives us only part of the picture.\n\nWe can create reusable evaluation datasets and testing patterns for common agent behaviors.\n\nThis is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified.\n\nDefine what can and cannot be captured by default.\n\nThe shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads.\n\nThis should include clear ownership, access policies, retention, and audit requirements.\n\nA useful pilot would be a multi-agent workflow that involves several independently deployed services.\n\nFor example, a merge request review involving coordinator, security, and testing agents.\n\nThe pilot should demonstrate:\n\nThis would provide a reference architecture that other teams can reuse.\n\nObservability should not begin after production deployment.\n\nWe can introduce telemetry and evaluation requirements into the development process itself.\n\nThis makes observability a standard engineering practice rather than a separate monitoring task.\n\nI would avoid introducing every observability component at once.\n\nA phased approach is more useful.\n\n| Phase | Focus | Expected result | \n|---|---|---|\n| 1 | OpenTelemetry initialization | Basic traces and metrics | \n| 2 | AI SDK instrumentation | Model and tool visibility | \n| 3 | Custom agent spans | Complete workflow tracing | \n| 4 | Cost attribution | Usage and cost per task | \n| 5 | Evaluation | Offline and online quality checks | \n| 6 | Security and governance | Privacy and access controls | \n| 7 | Shared dashboards | Cross-team operational visibility | \n\nThe first milestone should be simple:\n\n**For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result.**\n\nOnce this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce.\n\nAI agents introduce a new level of complexity to application observability.\n\nA single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution.\n\nOpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components.\n\nCombined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production.\n\nBut the most important distinction remains:\n\n**Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it.**\n\nFor an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices.\n\nThe goal should not be to collect more telemetry.\n\nIt should be to make every production agent workflow understandable, measurable, and continuously improvable.", "url": "https://wpnews.pro/news/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production", "canonical_source": "https://dev.to/serifcolakel/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production-1ln", "published_at": "2026-11-02 18:10:12+00:00", "updated_at": "2026-11-02 18:23:19.524722+00:00", "lang": "en", "topics": ["ai-agents", "mlops", "ai-tools", "developer-tools", "artificial-intelligence"], "entities": ["OpenTelemetry", "GitLab"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production", "markdown": "https://wpnews.pro/news/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production.md", "text": "https://wpnews.pro/news/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production.txt", "jsonld": "https://wpnews.pro/news/observability-for-ai-agents-tracing-evaluation-and-cost-control-in-production.jsonld"}}