{"slug": "mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for", "title": "MCP Deep Dive, Part 10: When the Agent Feels Off — Debugging and Observability for MCP in Production", "summary": "A developer debugging an MCP-based agent in production found that traditional request-level logging fails to capture multi-turn agent failures. By implementing OpenTelemetry-based distributed tracing with a root span per agent run, they reduced incident triage from hours to minutes. The approach correlates spans across the host, model, and servers, and adds structured logs and metrics dimensioned by tool and tenant.", "body_md": "A web API fails loudly: a 500, a stack trace, an alert. An agent fails\n\nsoftly. It doesn't crash — it quietly takes six turns instead of two, calls the wrong tool, spends triple the tokens, and returns an answer that's subtly wrong. None of that throws an exception, and none of it shows up in the logs of any single request.\n\nThis is **Part 10 of a 15-part deep dive on Model Context Protocol (MCP)**. Part 3 gave each tool call a span on the server; this part joins those into the whole picture — one agent run across the host, the model, and all three servers, correlated so that when something feels off you have an answer instead of a shrug.\n\n```\nagent.run  (tenant, goal)                                       [==================] 1.8s\n|\n+- agent.turn 0                                                 [======]\n|    +- agent.model_call  (plan)                                [===]\n|    +- tool.get_campaign_kpis  -> mattrx-analytics             [==]   120ms\n|    +- tool.query_events       -> mattrx-analytics             [===]  180ms\n|\n+- agent.turn 1                                                 [=====]\n|    +- agent.model_call  (synthesize)                          [====]\n|    +- tool.create_report      -> mattrx-reports (enqueue)     [=]    90ms\n|\n+- eval gate 0.93 (pass) -> final answer\n```\n\nStart one root span per agent run and propagate its context across the MCP transport, so every server's spans nest under it.\n\n```\npublic async Task<AgentAnswer> RunAsync(AiPrincipal p, string goal, CancellationToken ct)\n{\n    using var run = ActivitySource.StartActivity(\"agent.run\");   // the root span for the whole run\n    run?.SetTag(\"mattrx.tenant\", p.TenantId);\n    run?.SetTag(\"agent.goal\", Redact(goal));\n\n    // OTel context propagates over the MCP transport (traceparent) -> every server span nests here.\n    return await LoopAsync(p, goal, ct);\n}\n```\n\nThe unit of observability for an agent is the **run**, not the request. One trace id, and the whole cross-server run is in front of you — which is why incident triage dropped from **hours to minutes**.\n\nA span per turn, per model call, and per tool call, nested under the run.\n\n``` js\nfor (var turn = 0; turn < MaxTurns; turn++)\n{\n    using var turnSpan = ActivitySource.StartActivity(\"agent.turn\");\n    turnSpan?.SetTag(\"agent.turn\", turn);\n\n    using (ActivitySource.StartActivity(\"agent.model_call\"))\n        reply = await model.ChatAsync(messages, tools, ct);           // one span per model call\n\n    foreach (var call in reply.ToolCalls)\n        using (var t = ActivitySource.StartActivity($\"tool.{call.Name}\"))\n        {\n            t?.SetTag(\"mcp.server\", Route(call.Name));\n            results.Add(await manager.InvokeAsync(call, ct));\n        }\n}\n```\n\nThe tool span answers \"was the tool slow?\"; the run->turn->call tree answers \"why did the agent take six turns?\"\n\nStructured logs of the MCP interaction — method, tool, server, tenant, latency, outcome — correlated by the trace id and **redacted**.\n\n```\nlogger.ToolCall(new\n{\n    TraceId    = Activity.Current?.TraceId.ToString(),\n    Tool       = call.Name, Server = Route(call.Name),\n    Tenant     = principal.TenantId,\n    DurationMs = sw.ElapsedMilliseconds,\n    Outcome    = result.IsError ? \"error\" : \"ok\",\n    // arguments/results: a redacted projection or a hash — never the raw content\n});\n```\n\nTraces give order and latency; logs give detail. Correlate them by trace id — and redact the payloads so your observability stack doesn't become your largest unsecured copy of customer data.\n\nMCP-specific metrics, **dimensioned by tool and tenant**: latency and error rate per tool, turns per run, tokens and cost per run.\n\n```\nmeters.ToolLatency.Record(sw.ElapsedMilliseconds,\n    [KeyValuePair.Create(\"tool\", call.Name), KeyValuePair.Create(\"tenant\", principal.TenantId)]);\nmeters.ToolErrors.Add(result.IsError ? 1 : 0, /* same tags */);\nmeters.TurnsPerRun.Record(turnCount);\nmeters.TokensPerRun.Record(usage.TotalTokens);\n```\n\n\"Requests per second\" tells you nothing about whether your agent is healthy. Per-tool per-tenant latency turns \"the assistant is slow\" into \"`query_events`\n\non tenant Y regressed at 14:00.\" Turns-per-run catches thrashing; tokens/cost-per-run catches the agent that quietly got expensive.\n\nThe append-only audit log you already built for security **reconstructs** exactly what the agent saw and did.\n\n``` php\nvar reconstruction = await audit.ReconstructAsync(traceId, ct);\n// -> every tool call, result hash, guardrail decision, and outcome, in sequence\n```\n\nAgents are non-deterministic, so \"just reproduce it\" usually fails. The audit trail is your reconstruction — an investigation becomes a query, not an archaeology dig.\n\nPoke the server directly, and replay a recorded session to reproduce a bug deterministically.\n\n```\n# Poke a server directly — list tools, call one, see the raw JSON-RPC request/response.\nnpx @modelcontextprotocol/inspector node ./mattrx-analytics-server\n// Replay a recorded session against a server to reproduce a bug without the model in the loop.\nawait replayer.RunAsync(\"session-4821.jsonl\", targetServer: \"mattrx-analytics\", ct);\n```\n\nThe Inspector is the fastest way to answer \"is it the server or the agent?\" — call the tool by hand, no model involved. Replaying a recorded session turns a bug that showed up once into a repeatable test.\n\n``` php\nSymptom                         Look at...\nslow                     ->  the RUN trace: which turn/tool span is fat?\nwrong answer             ->  the AUDIT log: what did it retrieve + call?\ntoo many turns / cost    ->  turns-per-run + tokens-per-run metrics\nintermittent failures    ->  per-tool per-tenant error rate + retries\n\"did the server change?\" ->  the MCP Inspector: call the tool by hand\ncan't reproduce          ->  REPLAY the recorded session against the server\n```\n\n**Agents fail softly, so you have to watch softly too.** The exception-and-alert model built for web APIs misses the slow drift, the extra turns, the subtly worse answer. Observe the whole run as one trace, keep an audit record you can reconstruct from, dimension your metrics by tool and tenant, and watch quality alongside latency. Then \"the agent feels off\" has an answer.\n\n*Originally published at prepstack.co.in. Part 11 zooms out: rolling MCP across the enterprise.*", "url": "https://wpnews.pro/news/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for", "canonical_source": "https://dev.to/kirandeepjassalcrypto/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for-mcp-in-production-1c43", "published_at": "2026-07-21 16:03:08+00:00", "updated_at": "2026-07-21 16:23:11.145510+00:00", "lang": "en", "topics": ["artificial-intelligence", "ai-agents", "developer-tools", "ai-infrastructure"], "entities": ["Model Context Protocol", "OpenTelemetry"], "alternates": {"html": "https://wpnews.pro/news/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for", "markdown": "https://wpnews.pro/news/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for.md", "text": "https://wpnews.pro/news/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for.txt", "jsonld": "https://wpnews.pro/news/mcp-deep-dive-part-10-when-the-agent-feels-off-debugging-and-observability-for.jsonld"}}