When Your Agents Go Dark: Observability in Multi-Agent Systems with OpenTelemetry

Multi-agent AI systems introduce distributed-system failure modes that traditional logging cannot solve, according to a technical article by Matteo Rossi. The author argues that OpenTelemetry, an open-source observability framework hosted by the Cloud Native Computing Foundation, provides the trace structure needed to debug agent handoffs and failures. The piece walks through instrumenting agent handoffs with OpenTelemetry and contrasts it with simpler logging approaches for single-agent loops.

Written by Matteo Rossi . In recent months, AI applications have radically evolved. Earlier, prototypes looked like a single loop: prompt, model call, optional tool call, response. In production, an agent often becomes a small system: model calls, tool calls, routing logic, memory, retries, and error handling. We can have an agent that acts as a router: delegating tasks to other specialized agents; a retrieval agent that fetches information from databases and sends it to an agent responsible for reasoning; or an agent that critically reviews the work of other agents. Each of these agents will have its own model calls, its own reasoning and execution logic, and at each of these points, a potential point of failure and an error-handling mechanism. The parallel with traditional systems is more applicable today than ever. Once two agents coordinate through a handoff, a queue, a tool call, or shared state, the system has distributed-system failure modes. That means the debugging problem changes. The hard part is no longer fixing the failing step; it is finding which step failed and why. Years of production incidents have given us an operational model built on logs, metrics, traces, and alerts tied to expected service behavior. For new AI applications that use agents, we often have the same tools, but not the trace structure or attributes needed to answer the question: what happened and why? Logs can show local events, but they do not show the request path unless those events are correlated into a trace. The standard for producing those correlated traces is OpenTelemetry https://opentelemetry.io/ OTel : an open-source, vendor-neutral observability framework hosted by the Cloud Native Computing Foundation https://www.cncf.io/ that defines a common way to generate and export telemetry. It gives us a protocol for shipping data OTLP and a shared vocabulary of attribute names and semantic conventions, so that a span means the same thing across tools. The rest of this article applies it to multi-agent systems. This article walks that path in order: the failure modes that logs miss, how to instrument an agent handoff in code, what the resulting trace looks like in a real run, and when the simpler approach is still the right one. Let’s take it one step at a time to understand what we’ve done so far and the main issues with this approach. This is essential for us to understand how we can move toward full observability https://www.mongodb.com/resources/basics/observability/?utm campaign=devrel&utm source=Third+Party+Content&utm medium=Call+to+Action+Link&utm content=mongodb-blog-publication-Towards-AI&utm term=marissa.doherty . The first agent that most teams develop consists of a single loop: a prompt, a call to a model, a call to a tool if needed , and a response. Debugging in this scenario is straightforward: log the prompt, log the calls to the tools and the model, and finally, the response. If something goes wrong along the way, we can review all the logs linearly and figure out where the error is. Everything lives within the same process, like a traditional monolithic application. For an agent this simple, structured logging is enough, and the tracing infrastructure described later would add cost with little return. The system grows and becomes more complex. Instead of a single, all-purpose agent, we now have a router that delegates tasks to specialized agents. In addition to the various specialized agents, we need to add a memory management service to persist the context and an orchestrator to manage everything consistently. This is a representative example of a common orchestrator pattern, where a router delegates to specialized agents, rather than a specific product. In a scenario like this, each box shown in the diagram logs to its own standard output, and execution is no longer linear. Sometimes control passes to an agent that returns its response. At other times, two or more agents may work in parallel. The earlier mental model of reading logs backward no longer holds because the request path crosses agent boundaries and may branch. When five or more agents each generate their own logs, we end up with a timeline of uncorrelated events. Reconstructing the path of a single user request means stitching events across services, comparing timestamps, and guessing which call belongs to which step. Parallel calls make timestamps unreliable, too. This holds whether a person or an automated, agentic log-analysis tool performs the reading. In both cases, the events lack a trace ID, so any reader must infer the request path. The cost of this shortcoming can be measured using one of the DORA https://dora.dev/guides/dora-metrics/ DevOps Research and Assessment metrics: Mean Time To Resolve MTTR . A small diagnosis task can become a multi-hour reconstruction exercise when most of the effort is spent piecing together and connecting the dots of what happened rather than resolving the problem. 2. The Causality Problem Even when we find the logs relevant to the incident, we only have part of the story. We know what happened, but we don’t always know why. We can see that an agent returns a particular response. Still, we cannot determine if what we receive is generated by a truncated context window, a retrieved document that was actually deprecated, or a threshold set too generously, allowing the model to hallucinate. Because execution paths can vary, traces need to capture the decisions made along the way: tool choice, arguments, retrieved sources, model parameters, and token use. The same input can produce two different execution paths, depending on what the agent decides to do in a fully autonomous, non-deterministic manner. 3. The Cost Problem The cost principle for an agent is very simple: every agent interaction is a model call, and every model call consumes tokens. In a single-agent system, calculating the cost is straightforward because there is a single stream of requests associated with a single cost account. In a multi-agent system, however, a single user request can trigger an indefinite number of agent actions, with subsequent model calls, each of which may affect the responses provided by the LLM and potentially trigger further calls. A common pattern amplifies this: in an agentic evaluation loop, an agent reviews its own answer or the evidence supporting it and retries until it is satisfied. The result is a multitude of model calls that consume a high number of tokens. Attributing that cost back to the agent that generated it becomes difficult. 4. The latency problem Latency is another concern. In multi-agent systems, there are numerous potential sources of delay: serialization/deserialization of responses or calls, a slow-responding model, or a poorly configured timeout. When responses slow down, span timing shows whether time is spent on retrieval, model calls, tool execution, serialization, or waiting on another agent. We have seen a version of this before. It is the microservices observability problem in new clothing. That is good news because distributed tracing provides a useful base, and OpenTelemetry is the standard way to model and export those traces. Agent orchestration adds GenAI-specific attributes on top: OpenTelemetry now includes semantic conventions for Generative AI https://github.com/open-telemetry/semantic-conventions-genai . It records model calls, token usage, prompts, completions, tools, and related events. This standardized vocabulary includes various attributes such as the model name, the number of tokens used, and the type of operation performed. That keeps the instrumentation portable across open-source and proprietary backends. Each agent emits a span under the orchestrator span. The application exports them over OTLP to an OpenTelemetry collector that forwards traces to a trace backend and metrics to a metrics backend. The trace backend reassembles the spans into a single trace per request, so the full causal path is visible in one place. Building on those tracing concepts, we can now instrument an actual agent handoff in code. The principle behind this strategy is straightforward: wrap every meaningful unit of work within a span, and track the relevant information about that specific unit within the span’s attributes. Let’s illustrate this situation with an example, using an agent written in Java and instrumented with the OpenTelemetry APIs. In this example, the agent is delegating the task to a specialized agent. Span orchestratorSpan = tracer.spanBuilder "agent.orchestrator" .setSpanKind SpanKind.INTERNAL .startSpan ;try Scope scope = orchestratorSpan.makeCurrent { // Child span for the model call, following GenAI semantic conventions Span policySpan = tracer.spanBuilder "chat policy-agent" .setSpanKind SpanKind.CLIENT .setAttribute "gen ai.operation.name", "chat" .setAttribute "gen ai.system", "openai" .setAttribute "gen ai.request.model", "gpt-4o" .setAttribute "gen ai.request.temperature", 0.2 .startSpan ;try Scope policyScope = policySpan.makeCurrent { PolicyResult result = policyAgent.evaluate context ; // Record the decision and the cost policySpan.setAttribute "gen ai.usage.input tokens", result.promptTokens ; policySpan.setAttribute "gen ai.usage.output tokens", result.completionTokens ; policySpan.setAttribute "agent.policy.version", result.policyVersion ; policySpan.setAttribute "agent.retrieval.doc ids", result.sourceDocIds ; } catch Exception e { policySpan.recordException e ; policySpan.setStatus StatusCode.ERROR ;throw e;} finally { policySpan.end ;}} finally { orchestratorSpan.end ;} From this example, we can see that the child span is created while the orchestrator’s current span is active, which allows OpenTelemetry to link the two spans within the same trace. A note on versions: the OpenTelemetry semantic conventions for Generative AI are still in Development status and may change between releases. The attribute names used here follow version 1.42, pinned in the companion repository. Confirm the current version before reusing these names. Some attributes are still being renamed or relocated. It is not necessary to record every attribute included in the convention, though it is worth maintaining a minimal subset that serves as a baseline. In this way, we can transform diagnostic traces into something more meaningful and useful for our purposes. Many of these attributes do not need to be set manually when using Spring AI https://spring.io/projects/spring-ai . Spring AI is the Spring team’s framework for building AI applications in Java that wraps model providers with a common set of abstractions. For observability, it builds on Micrometer, the metrics and tracing facade used across the Spring ecosystem. Micrometer bridges to OpenTelemetry, so the data it produces can be exported over OTLP. On Spring AI 1.0, add two dependencies to the pom.xml spring-boot-starter-actuator plus a Micrometer-OTel bridge, such as micrometer-tracing-bridge-otel . The framework records Micrometer observations for supported model calls gen ai.client.operation and tool invocations spring.ai.tool , along with token-usage metrics gen ai.client.token.usage , without manually wrapping each call in tracing code. Those observations become spans and metrics through Micrometer’s bridge to OpenTelemetry. The manual spans created above, on the other hand, serve to show the framework what it cannot see: the orchestration layer. Spring AI does not know the decisions the orchestrator makes between model calls. It doesn’t know which routing logic was chosen or why a specific agent was called rather than another. These decisions lie within the application’s business logic and not in the framework, so they require explicit instrumentation. Jaeger https://www.jaegertracing.io/ is an open-source distributed tracing backend, hosted by the Cloud Native Computing Foundation. It collects spans and displays them as a trace tree. We use it here to see what the instrumentation produces. We export the agent’s spans to Jaeger over OpenTelemetry and run a sample orchestration. Any OpenTelemetry-compatible backend would work the same way. Jaeger is convenient because it is open-source and quick to run locally. Let’s see how a single orchestration trigger call gives rise to multiple spans, some of which contain calls to models in this case, Claude Haiku . The left column shows the trace tree. At the beginning is the incoming HTTP call, followed by the span related to the agent.orchestrator . It’s split into three child spans: chat-retrieval-agent , chat-policy-agent , and chat-summarizer-agent . Each of these is further split by Spring AI’s auto-instrumentation into a spring ai client call and an outbound call to the model. The timing bar shows the sequence of operations: retrieval, then policy application, then summarization. Within each trace, we can view the captured attributes according to OpenTelemetry’s conventional semantics. Here, gen ai.system identifies Anthropic as the provider, while gen ai.request.model and gen ai.response.model give the requested and returned model names, and gen ai.usage.input tokens 140 along with gen ai.usage.output tokens 59 report the cost per span. Finally, gen ai.response.finish reason indicates that the call completed successfully without errors. This trace shows which agent spent the tokens, which documents were retrieved to inform the policy decision, and where latency accumulated. This setup is not free, and it introduces dependencies: a collector to install, manage, and update; a backend to store traces; and a small amount of latency and compute overhead within each instrumented agent. As noted earlier, a single agent with a single prompt call and a couple of tools is well served by structured logging, and the setup described here is not worth its cost there. The same reasoning applies to internal prototypes with few users operating in purely experimental contexts. The issue is always complexity and the number of moving parts. The goal is to have observability strategies suited to the context, not ones that are complex because best practices dictate so. OpenTelemetry is not an observability product, but rather a protocol and a vocabulary. The next decision is whether to build the tracing layer ourselves or buy an observability platform. The market is currently full of observability platforms that now support observability for interactions with LLMs. To make informed choices, it is necessary to understand the fundamental difference between value and vendor lock-in for each tool. All of the platforms below interoperate with OpenTelemetry. What differs is how each engages with it; that difference shapes the lock-in. The table below summarizes all of these aspects. The underlying principle remains the same in every case. Choose the preferred platform, but ensure instrumentation follows open standards. If the spans that make up the trace follow conventional GenAI semantics and export metrics via OTLP, we can switch backends, change LLMs, or add specific tools to ensure observability without rewriting the core instrumentation model. The shared vocabulary is what makes this work. Once spans follow the GenAI conventions, any OTLP backend reads them identically. A multi-agent system has many of the same observability problems as a distributed system, with the added variability of model outputs, tool behavior, and dynamic routing. These systems operate according to fan-out strategies that are unknown in advance and make decisions based on the outputs they receive. All of this makes it practically impossible to rely solely on a log for observability, no matter how well-formatted or well-written it is. Therefore, the goal is to address this problem by collecting traces and metrics and storing them in tools that use a universal vocabulary. Whether we choose an open-source backend or a complete SaaS platform, the principle to apply remains the same. Capture all relevant traces and maintain an open, interoperable vocabulary and standard. The accompanying Java sample https://github.com/matteoroxis/agent-observability shows the trace setup, custom orchestration spans, and exported Jaeger trace. When Your Agents Go Dark: Observability in Multi-Agent Systems with OpenTelemetry https://pub.towardsai.net/when-your-agents-go-dark-observability-in-multi-agent-systems-with-opentelemetry-5454668c6360 was originally published in Towards AI https://pub.towardsai.net on Medium, where people are continuing the conversation by highlighting and responding to this story.