# Fixing X-Ray Service Maps for Multi-Agent Systems on AWS

> Source: <https://dev.to/joy_kareko_1cc956e11d827c/fixing-x-ray-service-maps-for-multi-agent-systems-on-aws-3glc>
> Published: 2026-11-13 19:30:38+00:00

I built a multi-agent system on Amazon Bedrock AgentCore. An orchestrator agent takes each request and hands the work to specialist worker agents: one checks orders, one handles refunds, one writes the reply to the customer. Some workers also search knowledge bases, a few of them in parallel.

Every agent worked. But the first time I opened the X-Ray service map, it showed the orchestrator, a box for Bedrock, and nothing else. No workers. No knowledge bases. Just a map that made my system look far simpler than it was.

Getting a map that matched the real call chain took me much longer than building the agents. This post is everything I wish I'd known before I started.

In this post:

A **trace** is the story of one request. When a customer asks for a refund, the trace records everything that happens because of it: the model calls, the tool calls and the work each agent does.

A trace is made of **spans**. A span is one step in the story, such as "asked the refund agent" or "searched the policy knowledge base". Each span records how long it took and whether it failed.

A **service map** turns traces into a picture. Each service is a box (a **node**), and each call between them is a line. For a multi-agent system, it's the quickest way to see which agent is slow, which one failed, and whether a request took the route you expected.

Here's the part that caught me out: **the map only draws what your traces describe as separate services.** If your code doesn't say "this is a call to a different thing", X-Ray has no reason to draw a separate box for it.

AgentCore sends traces through CloudWatch. Nothing appears until Transaction Search is on. You only do this once per AWS account and region.

In the CloudWatch console:

Then give it about ten minutes before you expect to see anything.

AgentCore uses **OpenTelemetry**, an open standard for recording traces, through the AWS Distro for OpenTelemetry (ADOT).

The setup depends on where your agent runs:

`requirements.txt`, and the runtime instruments your code when it starts. You don't need to run anything extra.

```
aws-opentelemetry-distro
strands-agents[otel]
```

If you run the system on your own machine while testing, start it like this:

```
export AGENT_OBSERVABILITY_ENABLED=true
export OTEL_PYTHON_DISTRO=aws_distro
export OTEL_PYTHON_CONFIGURATOR=aws_configurator

opentelemetry-instrument python agent.py
```

Two separate settings decide whether you see a test request, and it's easy to mix them up:

With a low sampling rate and only a handful of test requests, yours may never be traced, and it looks as if nothing happened.

While you're developing, set your sampling rate to 100% so every request is captured. You can keep a lower indexing percentage if you like. Before production, review both settings, because tracing and indexing more data costs more.

With all of this in place, you'll see traces. But you may still get the "orchestrator plus Bedrock" map I got. Here's why.

A common multi-agent pattern is **agents as tools**: each worker agent is wrapped in a tool, and the orchestrator calls the tool when it wants that worker's help.

``` php
@tool
def ask_refund_agent(request: str) -> str:
    """Send refund requests to the refund agent."""
    return str(refund_agent(request))
```

It's a clean design. The catch is that the worker runs **inside the orchestrator's process**. To the tracing library, it's just more code running in the same service. The worker's model calls show up, but they're filed under the orchestrator. There's no separate refund agent box.

**The fix: wrap each worker call in its own named span, and mark it as a call to something downstream.**

Two details matter:

`CLIENT`, which means "I'm calling something else".` peer.service`
Be clear about what this does. Your workers are still running inside the orchestrator's process, so they're logical parts of one program rather than separate services. These spans make each worker call visible and named in the trace. In X-Ray they're recorded as downstream calls, which is usually what puts a separate, named box on the map. How exactly the map draws them depends on how X-Ray interprets the exported spans, so check the result against your own map.

Here's a small helper:

``` python
from opentelemetry import trace
from opentelemetry.trace import SpanKind, Status, StatusCode

tracer = trace.get_tracer("orchestrator")

def traced_call(node_name, fn):
    """Run fn as a call to a separate node on the service map."""
    with tracer.start_as_current_span(
        node_name,
        kind=SpanKind.CLIENT,
        attributes={"peer.service": node_name},
    ) as span:
        try:
            return fn()
        except Exception as exc:
            span.record_exception(exc)
            span.set_status(Status(StatusCode.ERROR))
            raise
```

And here's the tool using it:

``` php
@tool
def ask_refund_agent(request: str) -> str:
    """Send refund requests to the refund agent."""
    return traced_call("refund-agent", lambda: str(refund_agent(request)))
```

Do the same for every worker, with a clear and different name for each one: `orders-agent`, `refund-agent`, `reply-agent`. Each worker call now has its own name in the trace, and a failure is recorded against the right worker.

Knowledge base searches use the AWS SDK. When AWS SDK instrumentation is enabled (ADOT does this for `boto3`), these calls can appear in your traces. But every search goes to the same AWS service, so they can be hard to tell apart. If you search three knowledge bases, you can't easily see which one is slow or failing.

**The fix is the same idea:** put a named span around each search, so you can see exactly which knowledge base was queried.

``` python
bedrock_agent_runtime = boto3.client("bedrock-agent-runtime")

def search_kb(kb_id, kb_name, query):
    return traced_call(
        f"KnowledgeBase:{kb_name}",
        lambda: bedrock_agent_runtime.retrieve(
            knowledgeBaseId=kb_id,
            retrievalQuery={"text": query},
        ),
    )
```

Now each knowledge base gets its own node, such as `KnowledgeBase:returns-policy` and `KnowledgeBase:shipping-policy`.

To save time, I searched several knowledge bases at once using a thread pool. Afterwards, the parallel searches didn't show up under the request at all. Some appeared as small traces of their own.

Here's why. OpenTelemetry keeps track of "which trace am I in?" in something called the **context**. A new thread starts with an empty context by default. So each search started a brand-new trace instead of joining the request's trace.

**The fix: copy the context into each thread.**

``` python
import contextvars
from concurrent.futures import ThreadPoolExecutor

KNOWLEDGE_BASES = [
    ("KB_ID_1", "returns-policy"),
    ("KB_ID_2", "shipping-policy"),
    ("KB_ID_3", "premium-benefits"),
]

with ThreadPoolExecutor() as pool:
    futures = [
        pool.submit(contextvars.copy_context().run, search_kb, kb_id, kb_name, query)
        for kb_id, kb_name in KNOWLEDGE_BASES
    ]
    results = [future.result() for future in futures]
```

`contextvars.copy_context()` takes a copy of the current context, including the trace, and `.run` starts the search inside that copy. Notice that the list makes a **new copy for each task**. Keep it that way: don't make one copy and share it between threads.

Each parallel search now joins the same trace, and the map shows the knowledge bases branching off together.

If some of your agents run in **separate** AgentCore runtimes rather than as tools, a different problem appears. Each agent shows up, but as an island with no line between them.

The caller has to pass the trace along in the request headers (`traceparent` or `X-Amzn-Trace-Id`). If it doesn't, the receiving agent starts a new trace.

`boto3` (`invoke_agent_runtime`), the instrumentation adds the header for you.

``` python
import requests
from opentelemetry.propagate import inject

headers = {"Content-Type": "application/json"}
inject(headers)  # adds the current trace to the headers

requests.post(WORKER_URL, json=payload, headers=headers)
```

`inject()` writes the header formats your **global propagator** is set up for. ADOT configures this for you. If you set up OpenTelemetry yourself, make sure the propagators match what the receiving side expects, for example `OTEL_PROPAGATORS=tracecontext,xray` for both `traceparent` and `X-Amzn-Trace-Id`.

You should see the orchestrator in the middle, a named connection to each worker agent, and each knowledge base search called out separately. Open a single trace too. One request should be one trace, with every worker and knowledge base search inside it.

*[Screenshot: service map before the fixes]*

*[Screenshot: service map after the fixes]*

The map also works as a test. If a simple question shows a line to the refund agent, your routing is wrong, not your tracing.

`opentelemetry-instrument`.
A service map can only draw what your traces describe. Auto-instrumentation records a lot, but it can't know that a function call is really "the refund agent", or that three searches belong together.

So if your map looks simpler than your system, it doesn't mean your system is broken. It means you need to tell the tracer where the boundaries are: name each agent, name each knowledge base, and carry the trace along wherever your code runs.

Have you fought with traces in your own multi-agent system? Tell me what tripped you up in the comments.

*I drafted this post with help from an AI assistant, then tested the steps and edited it myself.*