# Building a Production-Grade Kubernetes MCP Server with Dynatrace Observability

> Source: <https://blog.devgenius.io/building-a-production-grade-kubernetes-mcp-server-with-dynatrace-observability-7edde6a0961d?source=rss----4e2c1156667e---4>
> Published: 2026-10-26 11:20:37+00:00

I enabled AI assistants to access **Kubernetes** directly for network debugging and to correlate with [**Dynatrace**](https://www.dynatrace.com/). The solution uses TypeScript, avoids kubectl subprocesses, and self-monitors via [** OpenTelemetry**](https://opentelemetry.io/).

Roughly 60% of Kubernetes production incidents are caused by networking or DNS failures. Examples include NetworkPolicy blocking traffic, CoreDNS being OOMKilled, or a service selector becoming misaligned after a deployment rename. While the Kubernetes API contains the necessary diagnostic data, correlating it quickly under pressure remains a challenge.

I developed an [**MCP server**](https://modelcontextprotocol.io/docs/2026-07-28/learn/server-concepts) that gives AI assistants direct access to Kubernetes and Dynatrace, cutting correlation time from minutes to seconds.

**Model Context Protocol** is an open standard from **Anthropic** that acts as a universal connector for AI tools, similar to USB-C. Previously, each AI integration required custom code for each model and client. MCP standardizes the handshake, enabling a single server to support Claude Desktop, Claude Code, Cursor, and other compatible clients.

Instead of manually copying kubectl output into a chat window, the AI interacts directly with your tools and analyzes live data. You ask a question, and the AI investigates in real time.

***User: “Why is my checkout service timing out?”***
*Claude calls check_service_endpoints, finds no ready endpoints.
Claude calls trace_service_to_pods, finds selector mismatch.
Claude: “The checkout service has no ready endpoints because the pod selector does not match. Here is the fix.”
Total resolution time: 8 seconds.*

The server includes 20 tools across three core areas, all integrated with a non-invasive OpenTelemetry middleware layer:

Each tool invocation generates an OpenTelemetry trace, an invocation counter, and a latency measurement, all sent automatically to Dynatrace. The tool implementations contain no telemetry code.

Most Kubernetes tools rely on shelling out to kubectl, which introduces shell injection risks. For example, a pod name such as production-pod; rm -rf / can become a threat vector if interpolated into a shell command.

This server uses [**@kubernetes/client-node**](https://www.npmjs.com/package/@kubernetes/client-node) exclusively, so it makes direct, TLS-authenticated API calls without relying on a shell. TypeScript enforces argument types at compile time, and API calls execute in parallel:

```
const [podsRes, eventsRes] = await Promise.all([  coreV1Api.listNamespacedPod(namespace),  coreV1Api.listNamespacedEvent(namespace, undefined,    undefined, undefined, 'type=Warning'),]);coreV1Api.listNamespacedPod(namespace),
```

The overall response time is determined by the slowest call, rather than the sum of all calls.

Observability is essential for every tool, but embedding telemetry code within each tool increases coupling and maintenance overhead. Instead, I applied the decorator pattern.

One function wraps every tool:

``` js
export function withTelemetry(toolName, handler) {  return async (args) => {    return tracer.startActiveSpan(`mcp.tool.${toolName}`, async (span) => {      invocationCounter.add(1, { tool: toolName });      try {        const result = await handler(args);        span.setStatus({ code: SpanStatusCode.OK });        return result;      } catch (err) {        errorCounter.add(1, { tool: toolName });        span.setStatus({ code: SpanStatusCode.ERROR });        throw err;      } finally { span.end(); }    });  };}return async (args) => {
```

In index.ts, every tool registration is one line:

```
handler: withTelemetry('list_pods', listPods),
```

Tool files contain only business logic, with no telemetry code, making them fully testable in isolation.

Dynatrace is the preferred backend for three reasons. It natively accepts OpenTelemetry via **/api/v2/otlp** without requiring a Collector. **Davis AI** automatically correlates anomalies across traces, metrics, and logs into unified problems.

Additionally, the entities accessed by the MCP server (services, pods, deployments) are often already monitored in Dynatrace, enabling cross-linking from tool traces to the entity’s full observability view.

The three Dynatrace tools close the loop between Kubernetes state and business impact:

Each tool call generates a distributed trace named mcp.tool.{toolName}, including duration, success status, and error details. Logs include trace IDs, allowing direct navigation from a log entry to the full trace for that request.

The system automatically collects three metrics: invocations per tool, errors per tool, and latency histograms per tool. Dashboards show the most-used tools, P95 latency per tool, and error-rate spikes that may indicate RBAC changes or connectivity issues before they affect users.

```
git clone https://github.com/mrafay7655/kubernetes-observability-mcp-server.gitcd kubernetes-observability-mcp-servernpm install && cp .env.example .envnpm run build
```

Add to Claude Desktop config:

```
{  "mcpServers": {    "kubernetes-observability": {      "command": "node",      "args": ["/path/to/dist/index.js"],      "env": {        "DT_ENV_URL": "https://YOUR_ENV.live.dynatrace.com",        "DT_API_TOKEN": "dt0c01.YOUR_TOKEN"      }    }  }}
```

Restart Claude Desktop and ask: “List pods in the default namespace.” The AI will query the cluster directly.

Kubernetes incidents are challenging not because the data is hidden, but because it lives in multiple places and must be manually correlated under pressure. MCP removes this bottleneck. The AI handles correlation, so you can focus on resolving the issue.

This server implements that concept for Kubernetes and Dynatrace. The patterns of native API access, the withTelemetry() decorator, and self-monitoring via OpenTelemetry apply to any domain where an AI assistant could benefit from direct access to live operational data.

The code is open source at [github.com/mrafay7655/kubernetes-observability-mcp-server](https://github.com/mrafay7655/kubernetes-observability-mcp-server). If you run Kubernetes in production, consider trying it. If a tool is missing, please open an issue.

**Muhammad Rafay** is a Cloud Security Engineer specializing in vulnerability management, immutable infrastructure, and DevSecOps architectures. He works with engineering teams to build resilient, automated pipelines that address emerging security threats while maintaining uptime.

[Building a Production-Grade Kubernetes MCP Server with Dynatrace Observability](https://blog.devgenius.io/building-a-production-grade-kubernetes-mcp-server-with-dynatrace-observability-7edde6a0961d) was originally published in [Dev Genius](https://blog.devgenius.io) on Medium, where people are continuing the conversation by highlighting and responding to this story.