Fixing X-Ray Service Maps for Multi-Agent Systems on AWS

A developer building a multi-agent system on Amazon Bedrock AgentCore documented how to make AWS X-Ray service maps accurately reflect orchestrator-to-worker call chains, which initially collapsed into a single orchestrator-plus-Bedrock view. The fix involves wrapping each worker call in its own named span marked with span kind CLIENT and a peer.service attribute, after enabling CloudWatch Transaction Search and configuring OpenTelemetry via the AWS Distro for OpenTelemetry with 100% sampling during development.

I built a multi-agent system on Amazon Bedrock AgentCore. An orchestrator agent takes each request and hands the work to specialist worker agents: one checks orders, one handles refunds, one writes the reply to the customer. Some workers also search knowledge bases, a few of them in parallel. Every agent worked. But the first time I opened the X-Ray service map, it showed the orchestrator, a box for Bedrock, and nothing else. No workers. No knowledge bases. Just a map that made my system look far simpler than it was. Getting a map that matched the real call chain took me much longer than building the agents. This post is everything I wish I'd known before I started. In this post: A trace is the story of one request. When a customer asks for a refund, the trace records everything that happens because of it: the model calls, the tool calls and the work each agent does. A trace is made of spans . A span is one step in the story, such as "asked the refund agent" or "searched the policy knowledge base". Each span records how long it took and whether it failed. A service map turns traces into a picture. Each service is a box a node , and each call between them is a line. For a multi-agent system, it's the quickest way to see which agent is slow, which one failed, and whether a request took the route you expected. Here's the part that caught me out: the map only draws what your traces describe as separate services. If your code doesn't say "this is a call to a different thing", X-Ray has no reason to draw a separate box for it. AgentCore sends traces through CloudWatch. Nothing appears until Transaction Search is on. You only do this once per AWS account and region. In the CloudWatch console: Then give it about ten minutes before you expect to see anything. AgentCore uses OpenTelemetry , an open standard for recording traces, through the AWS Distro for OpenTelemetry ADOT . The setup depends on where your agent runs: requirements.txt , and the runtime instruments your code when it starts. You don't need to run anything extra. aws-opentelemetry-distro strands-agents otel If you run the system on your own machine while testing, start it like this: export AGENT OBSERVABILITY ENABLED=true export OTEL PYTHON DISTRO=aws distro export OTEL PYTHON CONFIGURATOR=aws configurator opentelemetry-instrument python agent.py Two separate settings decide whether you see a test request, and it's easy to mix them up: With a low sampling rate and only a handful of test requests, yours may never be traced, and it looks as if nothing happened. While you're developing, set your sampling rate to 100% so every request is captured. You can keep a lower indexing percentage if you like. Before production, review both settings, because tracing and indexing more data costs more. With all of this in place, you'll see traces. But you may still get the "orchestrator plus Bedrock" map I got. Here's why. A common multi-agent pattern is agents as tools : each worker agent is wrapped in a tool, and the orchestrator calls the tool when it wants that worker's help. php @tool def ask refund agent request: str - str: """Send refund requests to the refund agent.""" return str refund agent request It's a clean design. The catch is that the worker runs inside the orchestrator's process . To the tracing library, it's just more code running in the same service. The worker's model calls show up, but they're filed under the orchestrator. There's no separate refund agent box. The fix: wrap each worker call in its own named span, and mark it as a call to something downstream. Two details matter: CLIENT , which means "I'm calling something else". peer.service Be clear about what this does. Your workers are still running inside the orchestrator's process, so they're logical parts of one program rather than separate services. These spans make each worker call visible and named in the trace. In X-Ray they're recorded as downstream calls, which is usually what puts a separate, named box on the map. How exactly the map draws them depends on how X-Ray interprets the exported spans, so check the result against your own map. Here's a small helper: python from opentelemetry import trace from opentelemetry.trace import SpanKind, Status, StatusCode tracer = trace.get tracer "orchestrator" def traced call node name, fn : """Run fn as a call to a separate node on the service map.""" with tracer.start as current span node name, kind=SpanKind.CLIENT, attributes={"peer.service": node name}, as span: try: return fn except Exception as exc: span.record exception exc span.set status Status StatusCode.ERROR raise And here's the tool using it: php @tool def ask refund agent request: str - str: """Send refund requests to the refund agent.""" return traced call "refund-agent", lambda: str refund agent request Do the same for every worker, with a clear and different name for each one: orders-agent , refund-agent , reply-agent . Each worker call now has its own name in the trace, and a failure is recorded against the right worker. Knowledge base searches use the AWS SDK. When AWS SDK instrumentation is enabled ADOT does this for boto3 , these calls can appear in your traces. But every search goes to the same AWS service, so they can be hard to tell apart. If you search three knowledge bases, you can't easily see which one is slow or failing. The fix is the same idea: put a named span around each search, so you can see exactly which knowledge base was queried. python bedrock agent runtime = boto3.client "bedrock-agent-runtime" def search kb kb id, kb name, query : return traced call f"KnowledgeBase:{kb name}", lambda: bedrock agent runtime.retrieve knowledgeBaseId=kb id, retrievalQuery={"text": query}, , Now each knowledge base gets its own node, such as KnowledgeBase:returns-policy and KnowledgeBase:shipping-policy . To save time, I searched several knowledge bases at once using a thread pool. Afterwards, the parallel searches didn't show up under the request at all. Some appeared as small traces of their own. Here's why. OpenTelemetry keeps track of "which trace am I in?" in something called the context . A new thread starts with an empty context by default. So each search started a brand-new trace instead of joining the request's trace. The fix: copy the context into each thread. python import contextvars from concurrent.futures import ThreadPoolExecutor KNOWLEDGE BASES = "KB ID 1", "returns-policy" , "KB ID 2", "shipping-policy" , "KB ID 3", "premium-benefits" , with ThreadPoolExecutor as pool: futures = pool.submit contextvars.copy context .run, search kb, kb id, kb name, query for kb id, kb name in KNOWLEDGE BASES results = future.result for future in futures contextvars.copy context takes a copy of the current context, including the trace, and .run starts the search inside that copy. Notice that the list makes a new copy for each task . Keep it that way: don't make one copy and share it between threads. Each parallel search now joins the same trace, and the map shows the knowledge bases branching off together. If some of your agents run in separate AgentCore runtimes rather than as tools, a different problem appears. Each agent shows up, but as an island with no line between them. The caller has to pass the trace along in the request headers traceparent or X-Amzn-Trace-Id . If it doesn't, the receiving agent starts a new trace. boto3 invoke agent runtime , the instrumentation adds the header for you. python import requests from opentelemetry.propagate import inject headers = {"Content-Type": "application/json"} inject headers adds the current trace to the headers requests.post WORKER URL, json=payload, headers=headers inject writes the header formats your global propagator is set up for. ADOT configures this for you. If you set up OpenTelemetry yourself, make sure the propagators match what the receiving side expects, for example OTEL PROPAGATORS=tracecontext,xray for both traceparent and X-Amzn-Trace-Id . You should see the orchestrator in the middle, a named connection to each worker agent, and each knowledge base search called out separately. Open a single trace too. One request should be one trace, with every worker and knowledge base search inside it. Screenshot: service map before the fixes Screenshot: service map after the fixes The map also works as a test. If a simple question shows a line to the refund agent, your routing is wrong, not your tracing. opentelemetry-instrument . A service map can only draw what your traces describe. Auto-instrumentation records a lot, but it can't know that a function call is really "the refund agent", or that three searches belong together. So if your map looks simpler than your system, it doesn't mean your system is broken. It means you need to tell the tracer where the boundaries are: name each agent, name each knowledge base, and carry the trace along wherever your code runs. Have you fought with traces in your own multi-agent system? Tell me what tripped you up in the comments. I drafted this post with help from an AI assistant, then tested the steps and edited it myself.