MCP server architecture for platform teams

Anthropic's Model Context Protocol (MCP) is an open standard that enables AI agents to interact with external systems such as Kubernetes clusters, observability stacks, and ticketing systems, addressing AI hallucination and incident response by grounding AI answers in live system state. The article outlines production-grade MCP server architecture for platform engineering, emphasizing structured, auditable, and controlled access to live infrastructure, and notes that MCP alone does not solve tribal knowledge, which requires RAG alongside it.

⚡ Byte Size Summary - MCP Model Context Protocol is the standard that lets AI agents interact with external systems — your cluster, your observability stack, your ticketing system — without bespoke integration code for every tool. - MCP directly addresses AI hallucination and 2AM incident response by grounding AI answers in live system state. It doesn’t solve tribal knowledge alone — that needs RAG alongside it. - This article covers the production-grade architecture: what MCP servers are, how to design them for platform engineering use cases, and what you need to get right before running them anywhere near production. In Article 01 /posts/ai-tooling-openshift-evaluation-framework/ we built the evaluation framework. In Article 02 /posts/ai-in-the-stack-02-rag-runbooks/ we built a RAG pipeline for static runbook knowledge. Now we give AI agents access to live infrastructure. In logistics, the hardest problems rarely come from missing data. They come from disconnected systems. The warehouse knows one thing. The transportation management system knows another. Inventory systems lag behind reality by hours. Operators work around the gaps manually — copying numbers between screens, making calls to confirm what the system should already know, carrying context in their heads because no single system has the full picture. I spent years watching intelligent people solve problems that shouldn’t have existed, because the systems around them were designed to optimise locally rather than coordinate globally. The data was there. The capability was there. The coordination layer wasn’t. Modern infrastructure operations feel surprisingly similar. Your Kubernetes cluster knows the state of every pod. Your observability stack knows the error rates and latency trends. Your ticketing system knows what changes were deployed in the last 24 hours. Your CI/CD pipeline knows what’s currently in flight. And your AI assistant — the tool you’re increasingly asking to help you reason about incidents — knows none of it, unless you paste it in manually. Model Context Protocol is the coordination layer that changes this. Not by giving AI access to everything at once, but by giving it a structured, auditable, controlled way to request the context it needs, from the systems that have it, at the moment it needs it. That’s what this article is about. What MCP Actually Is what-mcp-actually-is Model Context Protocol MCP is an open standard, introduced by Anthropic, that defines how AI models communicate with external tools and data sources. Think of it as a common language that sits between an AI assistant and the systems it needs to interact with. Before MCP, every AI integration was bespoke. You wanted your LLM to query your Kubernetes cluster? Write a custom function. You wanted it to check PagerDuty? Write another one. You wanted it to search your runbooks and open a Jira ticket? Three separate integrations, all maintained independently, all breaking in different ways when APIs change. MCP replaces that with a standard. An MCP server exposes a set of tools — defined capabilities the AI can invoke — plus resources — data it can read. The AI client Claude, Cursor, any MCP-compatible host discovers what tools are available, decides which to call based on the user’s question, calls them, and incorporates the results into its response. The AI doesn’t have direct access to your systems. It has access to an MCP server that mediates that access. That distinction matters enormously for security and governance — which is why this article spends as much time on architecture as on implementation. Why Platform Engineers Should Care why-platform-engineers-should-care The RAG pipeline from Article 02 /posts/ai-in-the-stack-02-rag-runbooks/ was useful for static knowledge — runbooks, documentation, past incident reports. MCP is useful for live state. When an engineer asks “what is causing the latency spike in the payments service right now?” — that isn’t a runbook question. It requires current pod status, recent deployment events, live error rates, and possibly the last three alerts that fired. None of that lives in a document. All of it lives in systems your MCP server can reach. The distinction between what MCP solves and what it doesn’t matters before you design anything. AI hallucination — yes, directly. Hallucination happens when an LLM answers from training data instead of ground truth. MCP forces the AI to retrieve live, authoritative state before responding. It doesn’t eliminate hallucination entirely — an LLM can still misinterpret what it retrieves — but it directly attacks the root cause for infrastructure questions. 2AM incidents — yes, directly. This is the primary operational use case. Instead of an engineer manually checking five systems in sequence while half-asleep, an AI with MCP access can pull pod status, recent events, and active alerts in a single query and reason across all of it simultaneously. Speed and context at the moment they are hardest to find. Too many dashboards — partially. MCP doesn’t reduce the number of dashboards in your environment. It gives an AI a way to query across the systems those dashboards represent, so an engineer asks one question instead of navigating five screens. The dashboards still exist. You stop having to drive them manually during an incident. Tribal knowledge — not alone. MCP surfaces what your systems know. It doesn’t surface what your team knows — the undocumented context that lives in people’s heads, the runbook that exists nowhere in any system, the reason a service is named what it is. That’s a RAG problem. The combination of RAG for historical and human knowledge and MCP for live system state is where the tribal knowledge gap actually starts to close. Neither alone is sufficient. An AI that can read your runbooks and query your cluster simultaneously is a meaningful operational tool. An AI that can only do one of those things is a limited one. MCP Server Architecture for Platform Engineering mcp-server-architecture-for-platform-engineering A production-grade MCP server for a platform team has four layers: Every tool invocation travels this path: the AI client sends a request, the Auth Gateway validates identity before anything reaches your infrastructure, the MCP server processes it through governance and audit controls, and the Kubernetes API Server enforces access policy independently of the application layer. Two enforcement gates — not one. That’s the architecture the implementation sections below are built around. The four layers in code: Layer 1 — Governance First layer-1--governance-first Before writing a single tool definition, decide and enforce these three things: Read-only by default. Every tool that touches production infrastructure should be read-only unless you have explicitly designed the write path with human approval steps. An MCP server that can kubectl delete anything is an incident waiting to happen. Start with read, earn trust, expand deliberately. Audit logging. Every tool call should be logged with: timestamp, tool name, input parameters, calling session identity, and response status. This is your audit trail when something goes wrong. It’s also how you demonstrate to your security team that AI isn’t a black box. Rate limiting. An AI in an agentic loop can call tools hundreds of times in seconds. Without rate limiting, a runaway agent can exhaust your Kubernetes API quota, spam your ticketing system, or trigger alert storms in your observability stack. Set per-session and per-tool limits before you deploy. Layer 2 — Backend Clients layer-2--backend-clients The MCP server needs clients for each system it connects to. Keep these thin — their job is to call APIs and return structured data, not to contain business logic. For a Kubernetes-connected MCP server, using the official kubernetes Python client: python k8s client.py from kubernetes import client, config from typing import Optional class KubernetesClient: def init self, in cluster: bool = False : if in cluster: config.load incluster config else: config.load kube config self.v1 = client.CoreV1Api self.apps v1 = client.AppsV1Api def get pod status self, namespace: str, pod name: str - dict: pod = self.v1.read namespaced pod name=pod name, namespace=namespace return { "name": pod.metadata.name, "namespace": pod.metadata.namespace, "phase": pod.status.phase, "conditions": {"type": c.type, "status": c.status, "reason": c.reason} for c in pod.status.conditions or , "container statuses": { "name": cs.name, "ready": cs.ready, "restart count": cs.restart count, "state": str cs.state } for cs in pod.status.container statuses or } def list failing pods self, namespace: Optional str = None - list dict : if namespace: pods = self.v1.list namespaced pod namespace=namespace else: pods = self.v1.list pod for all namespaces failing = for pod in pods.items: if pod.status.phase not in "Running", "Succeeded" : failing.append { "name": pod.metadata.name, "namespace": pod.metadata.namespace, "phase": pod.status.phase, "reason": pod.status.reason } return failing def get recent events self, namespace: str, limit: int = 20 - list dict : events = self.v1.list namespaced event namespace=namespace, limit=limit return { "type": e.type, "reason": e.reason, "message": e.message, "involved object": e.involved object.name, "count": e.count, "last timestamp": str e.last timestamp } for e in sorted events.items, key=lambda x: x.last timestamp or "", reverse=True Layer 3 — Tool Definitions layer-3--tool-definitions This is the layer the AI interacts with directly. Tool descriptions aren’t just documentation — they’re what the LLM reads to decide whether to call the tool and how to format its inputs. Write them precisely. python tools.py from mcp.server import Server from mcp.types import Tool, TextContent import json import logging from k8s client import KubernetesClient from audit import log tool call logger = logging.getLogger name k8s = KubernetesClient in cluster=False Set True when running inside the cluster def register tools server: Server : @server.list tools async def list tools : return Tool name="get pod status", description= "Get the current status of a specific Kubernetes pod, including phase, " "readiness conditions, container states, and restart counts. " "Use this when investigating why a specific pod is unhealthy or not ready." , inputSchema={ "type": "object", "properties": { "namespace": { "type": "string", "description": "The Kubernetes namespace the pod is in" }, "pod name": { "type": "string", "description": "The exact name of the pod" } }, "required": "namespace", "pod name" } , Tool name="list failing pods", description= "List all pods that are not in Running or Succeeded state across the cluster " "or within a specific namespace. Use this as a first step when an incident " "is reported and you need to identify which pods are affected." , inputSchema={ "type": "object", "properties": { "namespace": { "type": "string", "description": "Optional: filter to a specific namespace" } } } , Tool name="get recent events", description= "Retrieve recent Kubernetes events for a namespace, ordered by most recent first. " "Events capture warnings, errors, and state changes. Use this to understand " "what happened in the cluster leading up to an issue." , inputSchema={ "type": "object", "properties": { "namespace": { "type": "string", "description": "The namespace to retrieve events from" }, "limit": { "type": "integer", "description": "Maximum number of events to return default 20 ", "default": 20 } }, "required": "namespace" } @server.call tool async def call tool name: str, arguments: dict : log tool call tool=name, inputs=arguments Always audit first try: if name == "get pod status": result = k8s.get pod status namespace=arguments "namespace" , pod name=arguments "pod name" elif name == "list failing pods": result = k8s.list failing pods namespace=arguments.get "namespace" elif name == "get recent events": result = k8s.get recent events namespace=arguments "namespace" , limit=arguments.get "limit", 20 else: return TextContent type="text", text=f"Unknown tool: {name}" return TextContent type="text", text=json.dumps result, indent=2 except Exception as e: logger.error f"Tool {name} failed: {str e }" return TextContent type="text", text=f"Tool execution failed: {str e }" Layer 4 — Transport and Auth layer-4--transport-and-auth MCP supports two transport modes: stdio — the server runs as a subprocess of the AI client. Simple, local, no network exposure. Right for developer workstations and local tooling. HTTP with SSE Server-Sent Events — the server runs as a persistent service, reachable over the network. Required for shared team tooling, remote access, and running inside a cluster. For production deployments, SSE transport with mutual TLS mTLS is the hardened path; API key authentication is acceptable for internal cluster traffic with network policy controls in place. For a platform team MCP server running on Kubernetes: python main.py import asyncio import logging from mcp.server import Server from mcp.server.sse import SseServerTransport from starlette.applications import Starlette from starlette.routing import Route from starlette.middleware import Middleware from starlette.middleware.base import BaseHTTPMiddleware from tools import register tools logging.basicConfig level=logging.INFO server = Server "platform-mcp" register tools server class APIKeyMiddleware BaseHTTPMiddleware : async def dispatch self, request, call next : api key = request.headers.get "X-API-Key" if api key = EXPECTED API KEY: Load from env, not hardcoded from starlette.responses import JSONResponse return JSONResponse {"error": "Unauthorised"}, status code=401 return await call next request transport = SseServerTransport "/messages" async def handle sse request : async with transport.connect sse request.scope, request.receive, request. send as streams: await server.run streams 0 , streams 1 , server.create initialization options app = Starlette routes= Route "/sse", endpoint=handle sse , middleware= Middleware APIKeyMiddleware Kubernetes Deployment kubernetes-deployment k8s/deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: platform-mcp-server namespace: platform-tools spec: replicas: 1 selector: matchLabels: app: platform-mcp-server template: metadata: labels: app: platform-mcp-server spec: serviceAccountName: platform-mcp-sa Read-only SA — see RBAC below containers: - name: mcp-server image: your-registry/platform-mcp:latest ports: - containerPort: 8080 env: - name: MCP API KEY valueFrom: secretKeyRef: name: platform-mcp-secrets key: api-key --- k8s/rbac.yaml apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: platform-mcp-reader rules: - apiGroups: "" resources: "pods", "events", "namespaces", "nodes" verbs: "get", "list", "watch" Read-only — no create, update, delete - apiGroups: "apps" resources: "deployments", "replicasets" verbs: "get", "list", "watch" --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: platform-mcp-reader-binding subjects: - kind: ServiceAccount name: platform-mcp-sa namespace: platform-tools roleRef: kind: ClusterRole name: platform-mcp-reader apiGroup: rbac.authorization.k8s.io The RBAC configuration enforces the governance constraint at the Kubernetes level — not just in application code. Even if a bug in the tool definitions allowed a write operation to reach the Kubernetes client, the service account has no permission to execute it. Defence in depth. Not one gate — two. What This Unlocks what-this-unlocks With a platform MCP server running, a Claude-powered assistant can handle questions like these using live cluster data: - “What pods are failing in the payments namespace right now?” → calls list failing pods - “Why did the checkout service restart three times this morning?” → calls get pod status + get recent events - “Is there anything unusual happening across the cluster before I deploy?” → calls list failing pods across all namespaces This is the coordination layer the opening story was pointing at. In logistics, the fix for disconnected systems was never better dashboards — it was a shared integration layer that let every system speak to every other system through a common protocol. MCP is that layer for AI and infrastructure. Combined with the RAG pipeline from Article 02, the same assistant can cross-reference live cluster state against your runbooks — returning answers grounded in documentation and informed by current reality simultaneously. That’s the operational use case MCP was built for. What to Build Next what-to-build-next The server in this article covers Kubernetes read operations. The natural extensions, covered in the GitHub repo https://github.com/agentic-devops/pipelineandprompts-labs/tree/main/ai-in-the-stack/03-mcp-for-kubernetes , are: - Prometheus integration — add a get metrics tool that queries PromQL Prometheus Query Language and returns current error rates and latency percentiles - PagerDuty integration — add get active incidents and get recent alerts tools - Write operations with human approval — a restart pod tool that creates a Jira ticket and waits for human sign-off before executing; this is the governance pattern that makes agentic write operations safe in production The write operation pattern — where the AI prepares an action, a human approves it, and the MCP server executes — is covered in Article 05 of this series. What’s Next whats-next Article 04 — Prompt Versioning in Production: Treat Prompts Like Infrastructure Artifacts /posts/prompt-versioning-ci-openshift/ System prompts are configuration. Changing them without version control, testing, or rollback strategy is the same mistake engineers made with infrastructure before Terraform existed. Next: how to version, test, and deploy prompts with the same discipline you apply to everything else in your stack.