A web API fails loudly: a 500, a stack trace, an alert. An agent fails
softly. It doesn't crash — it quietly takes six turns instead of two, calls the wrong tool, spends triple the tokens, and returns an answer that's subtly wrong. None of that throws an exception, and none of it shows up in the logs of any single request.
This is Part 10 of a 15-part deep dive on Model Context Protocol (MCP). Part 3 gave each tool call a span on the server; this part joins those into the whole picture — one agent run across the host, the model, and all three servers, correlated so that when something feels off you have an answer instead of a shrug.
agent.run (tenant, goal) [==================] 1.8s
|
+- agent.turn 0 [======]
| +- agent.model_call (plan) [===]
| +- tool.get_campaign_kpis -> mattrx-analytics [==] 120ms
| +- tool.query_events -> mattrx-analytics [===] 180ms
|
+- agent.turn 1 [=====]
| +- agent.model_call (synthesize) [====]
| +- tool.create_report -> mattrx-reports (enqueue) [=] 90ms
|
+- eval gate 0.93 (pass) -> final answer
Start one root span per agent run and propagate its context across the MCP transport, so every server's spans nest under it.
public async Task<AgentAnswer> RunAsync(AiPrincipal p, string goal, CancellationToken ct)
{
using var run = ActivitySource.StartActivity("agent.run"); // the root span for the whole run
run?.SetTag("mattrx.tenant", p.TenantId);
run?.SetTag("agent.goal", Redact(goal));
// OTel context propagates over the MCP transport (traceparent) -> every server span nests here.
return await LoopAsync(p, goal, ct);
}
The unit of observability for an agent is the run, not the request. One trace id, and the whole cross-server run is in front of you — which is why incident triage dropped from hours to minutes.
A span per turn, per model call, and per tool call, nested under the run.
for (var turn = 0; turn < MaxTurns; turn++)
{
using var turnSpan = ActivitySource.StartActivity("agent.turn");
turnSpan?.SetTag("agent.turn", turn);
using (ActivitySource.StartActivity("agent.model_call"))
reply = await model.ChatAsync(messages, tools, ct); // one span per model call
foreach (var call in reply.ToolCalls)
using (var t = ActivitySource.StartActivity($"tool.{call.Name}"))
{
t?.SetTag("mcp.server", Route(call.Name));
results.Add(await manager.InvokeAsync(call, ct));
}
}
The tool span answers "was the tool slow?"; the run->turn->call tree answers "why did the agent take six turns?"
Structured logs of the MCP interaction — method, tool, server, tenant, latency, outcome — correlated by the trace id and redacted.
logger.ToolCall(new
{
TraceId = Activity.Current?.TraceId.ToString(),
Tool = call.Name, Server = Route(call.Name),
Tenant = principal.TenantId,
DurationMs = sw.ElapsedMilliseconds,
Outcome = result.IsError ? "error" : "ok",
// arguments/results: a redacted projection or a hash — never the raw content
});
Traces give order and latency; logs give detail. Correlate them by trace id — and redact the payloads so your observability stack doesn't become your largest unsecured copy of customer data.
MCP-specific metrics, dimensioned by tool and tenant: latency and error rate per tool, turns per run, tokens and cost per run.
meters.ToolLatency.Record(sw.ElapsedMilliseconds,
[KeyValuePair.Create("tool", call.Name), KeyValuePair.Create("tenant", principal.TenantId)]);
meters.ToolErrors.Add(result.IsError ? 1 : 0, /* same tags */);
meters.TurnsPerRun.Record(turnCount);
meters.TokensPerRun.Record(usage.TotalTokens);
"Requests per second" tells you nothing about whether your agent is healthy. Per-tool per-tenant latency turns "the assistant is slow" into "query_events
on tenant Y regressed at 14:00." Turns-per-run catches thrashing; tokens/cost-per-run catches the agent that quietly got expensive.
The append-only audit log you already built for security reconstructs exactly what the agent saw and did.
var reconstruction = await audit.ReconstructAsync(traceId, ct);
// -> every tool call, result hash, guardrail decision, and outcome, in sequence
Agents are non-deterministic, so "just reproduce it" usually fails. The audit trail is your reconstruction — an investigation becomes a query, not an archaeology dig.
Poke the server directly, and replay a recorded session to reproduce a bug deterministically.
npx @modelcontextprotocol/inspector node ./mattrx-analytics-server
// Replay a recorded session against a server to reproduce a bug without the model in the loop.
await replayer.RunAsync("session-4821.jsonl", targetServer: "mattrx-analytics", ct);
The Inspector is the fastest way to answer "is it the server or the agent?" — call the tool by hand, no model involved. Replaying a recorded session turns a bug that showed up once into a repeatable test.
Symptom Look at...
slow -> the RUN trace: which turn/tool span is fat?
wrong answer -> the AUDIT log: what did it retrieve + call?
too many turns / cost -> turns-per-run + tokens-per-run metrics
intermittent failures -> per-tool per-tenant error rate + retries
"did the server change?" -> the MCP Inspector: call the tool by hand
can't reproduce -> REPLAY the recorded session against the server
Agents fail softly, so you have to watch softly too. The exception-and-alert model built for web APIs misses the slow drift, the extra turns, the subtly worse answer. Observe the whole run as one trace, keep an audit record you can reconstruct from, dimension your metrics by tool and tenant, and watch quality alongside latency. Then "the agent feels off" has an answer.
Originally published at prepstack.co.in. Part 11 zooms out: rolling MCP across the enterprise.