MCP Deep Dive, Part 10: When the Agent Feels Off — Debugging and Observability for MCP in Production

A developer debugging an MCP-based agent in production found that traditional request-level logging fails to capture multi-turn agent failures. By implementing OpenTelemetry-based distributed tracing with a root span per agent run, they reduced incident triage from hours to minutes. The approach correlates spans across the host, model, and servers, and adds structured logs and metrics dimensioned by tool and tenant.

A web API fails loudly: a 500, a stack trace, an alert. An agent fails softly. It doesn't crash — it quietly takes six turns instead of two, calls the wrong tool, spends triple the tokens, and returns an answer that's subtly wrong. None of that throws an exception, and none of it shows up in the logs of any single request. This is Part 10 of a 15-part deep dive on Model Context Protocol MCP . Part 3 gave each tool call a span on the server; this part joins those into the whole picture — one agent run across the host, the model, and all three servers, correlated so that when something feels off you have an answer instead of a shrug. agent.run tenant, goal ================== 1.8s | +- agent.turn 0 ====== | +- agent.model call plan === | +- tool.get campaign kpis - mattrx-analytics == 120ms | +- tool.query events - mattrx-analytics === 180ms | +- agent.turn 1 ===== | +- agent.model call synthesize ==== | +- tool.create report - mattrx-reports enqueue = 90ms | +- eval gate 0.93 pass - final answer Start one root span per agent run and propagate its context across the MCP transport, so every server's spans nest under it. public async Task<AgentAnswer RunAsync AiPrincipal p, string goal, CancellationToken ct { using var run = ActivitySource.StartActivity "agent.run" ; // the root span for the whole run run?.SetTag "mattrx.tenant", p.TenantId ; run?.SetTag "agent.goal", Redact goal ; // OTel context propagates over the MCP transport traceparent - every server span nests here. return await LoopAsync p, goal, ct ; } The unit of observability for an agent is the run , not the request. One trace id, and the whole cross-server run is in front of you — which is why incident triage dropped from hours to minutes . A span per turn, per model call, and per tool call, nested under the run. js for var turn = 0; turn < MaxTurns; turn++ { using var turnSpan = ActivitySource.StartActivity "agent.turn" ; turnSpan?.SetTag "agent.turn", turn ; using ActivitySource.StartActivity "agent.model call" reply = await model.ChatAsync messages, tools, ct ; // one span per model call foreach var call in reply.ToolCalls using var t = ActivitySource.StartActivity $"tool.{call.Name}" { t?.SetTag "mcp.server", Route call.Name ; results.Add await manager.InvokeAsync call, ct ; } } The tool span answers "was the tool slow?"; the run- turn- call tree answers "why did the agent take six turns?" Structured logs of the MCP interaction — method, tool, server, tenant, latency, outcome — correlated by the trace id and redacted . logger.ToolCall new { TraceId = Activity.Current?.TraceId.ToString , Tool = call.Name, Server = Route call.Name , Tenant = principal.TenantId, DurationMs = sw.ElapsedMilliseconds, Outcome = result.IsError ? "error" : "ok", // arguments/results: a redacted projection or a hash — never the raw content } ; Traces give order and latency; logs give detail. Correlate them by trace id — and redact the payloads so your observability stack doesn't become your largest unsecured copy of customer data. MCP-specific metrics, dimensioned by tool and tenant : latency and error rate per tool, turns per run, tokens and cost per run. meters.ToolLatency.Record sw.ElapsedMilliseconds, KeyValuePair.Create "tool", call.Name , KeyValuePair.Create "tenant", principal.TenantId ; meters.ToolErrors.Add result.IsError ? 1 : 0, / same tags / ; meters.TurnsPerRun.Record turnCount ; meters.TokensPerRun.Record usage.TotalTokens ; "Requests per second" tells you nothing about whether your agent is healthy. Per-tool per-tenant latency turns "the assistant is slow" into " query events on tenant Y regressed at 14:00." Turns-per-run catches thrashing; tokens/cost-per-run catches the agent that quietly got expensive. The append-only audit log you already built for security reconstructs exactly what the agent saw and did. php var reconstruction = await audit.ReconstructAsync traceId, ct ; // - every tool call, result hash, guardrail decision, and outcome, in sequence Agents are non-deterministic, so "just reproduce it" usually fails. The audit trail is your reconstruction — an investigation becomes a query, not an archaeology dig. Poke the server directly, and replay a recorded session to reproduce a bug deterministically. Poke a server directly — list tools, call one, see the raw JSON-RPC request/response. npx @modelcontextprotocol/inspector node ./mattrx-analytics-server // Replay a recorded session against a server to reproduce a bug without the model in the loop. await replayer.RunAsync "session-4821.jsonl", targetServer: "mattrx-analytics", ct ; The Inspector is the fastest way to answer "is it the server or the agent?" — call the tool by hand, no model involved. Replaying a recorded session turns a bug that showed up once into a repeatable test. php Symptom Look at... slow - the RUN trace: which turn/tool span is fat? wrong answer - the AUDIT log: what did it retrieve + call? too many turns / cost - turns-per-run + tokens-per-run metrics intermittent failures - per-tool per-tenant error rate + retries "did the server change?" - the MCP Inspector: call the tool by hand can't reproduce - REPLAY the recorded session against the server Agents fail softly, so you have to watch softly too. The exception-and-alert model built for web APIs misses the slow drift, the extra turns, the subtly worse answer. Observe the whole run as one trace, keep an audit record you can reconstruct from, dimension your metrics by tool and tenant, and watch quality alongside latency. Then "the agent feels off" has an answer. Originally published at prepstack.co.in. Part 11 zooms out: rolling MCP across the enterprise.