Building a Production AI Platform: Kubernetes, GitOps, IaC, Security, and Observability

A developer detailed the architecture of a production AI platform, emphasizing the integration of Kubernetes, GitOps, infrastructure as code, security, and observability. The post highlights that production systems require reproducible, secure, and observable infrastructure, with Kubernetes serving as the runtime for AI services, model servers, and agents. It also cites CNCF's 2025 survey showing 66% of organizations use Kubernetes for generative AI inference workloads.

Thanks for taking the time to read. If you’ve worked on AI platforms, cloud infrastructure, or platform engineering, I’d love to hear how your architecture differs in the comments. In Part 4 of AI Infrastructure for Cloud Engineers , we looked at FinOps for AI and how GPU utilization, token consumption, model choice, and inference volume affect cost. Read Part 4: FinOps for AI: Understanding GPU, Token, and Inference Costs So far, we have looked at individual parts of AI infrastructure: Kubernetes GPUs Scheduling Model Serving Observability FinOps But production systems rarely operate as separate pieces. The real challenge is bringing them together into a platform that developers can deploy to, operators can understand, security teams can govern, and businesses can afford to run. That is what this final article is about. A simplified architecture might look like this: Developer ↓ Git Repository ↓ CI Pipeline ↓ Container Registry ↓ GitOps Repository ↓ Kubernetes │ ├── AI Applications ├── Model Servers ├── GPU Workloads ├── Vector Services └── AI Agents │ ↓ Observability + Security + FinOps The model is only one component. A production AI platform also needs: The goal is not simply to make an AI application run. The goal is to make it repeatable, secure, observable, scalable, and recoverable . Creating infrastructure manually might work for an experiment. Production needs something reproducible. Instead of engineers manually creating: Kubernetes Cluster GPU Node Pool Network Storage Identity Secrets Integration Monitoring the infrastructure should be defined as code. Conceptually: Infrastructure Code ↓ Review ↓ Plan ↓ Apply ↓ Cloud Infrastructure This makes infrastructure: A typical repository might look like: infrastructure/ ├── network/ ├── kubernetes/ ├── gpu-nodes/ ├── identity/ ├── monitoring/ └── environments/ ├── dev/ ├── staging/ └── production/ The important principle is not the specific IaC tool. It is that infrastructure changes follow the same engineering discipline as application changes. Kubernetes becomes the runtime where the AI platform operates. It might host: AI API Model Server Embedding Service Vector Search AI Agents Background Workers GPU Workloads For example: Kubernetes │ ┌───────────────┼───────────────┐ ↓ ↓ ↓ AI Services Model Servers AI Agents │ │ │ └───────────────┼───────────────┘ ↓ GPU Pool This gives the platform a common way to manage: Scheduling Scaling Networking Health checks Rollouts Configuration Resource allocation Kubernetes adoption for AI is already moving in this direction. CNCF's 2025 survey found that 66% of organizations hosting generative AI models use Kubernetes for at least some inference workloads. The interesting part is that AI infrastructure begins to look less like a collection of individual servers and more like a shared platform. A useful pattern is to separate building software from deploying software . The CI pipeline can handle: Code ↓ Tests ↓ Security Scan ↓ Container Build ↓ Container Registry Deployment can then be handled separately through GitOps. Container Registry ↓ Deployment Configuration ↓ Git Repository ↓ GitOps Controller ↓ Kubernetes Why separate them? Because the CI system does not need broad credentials to modify production clusters directly. Instead, production configuration lives in Git. A change becomes something like: Pull Request ↓ Review ↓ Merge ↓ GitOps Reconciliation ↓ Deployment Kubernetes itself recommends declarative, version-controlled configuration for production workloads, which also fits naturally with GitOps workflows. Imagine production currently runs: model-version: v12 replicas: 4 A new release requires: model-version: v13 replicas: 6 Instead of manually changing the cluster, the team updates the configuration in Git. model: version: v13 replicas: 6 The GitOps controller compares: Desired State in Git ↓ Actual State in Cluster and reconciles the difference. That gives teams: Change history Code review Rollback Auditability Environment consistency If something goes wrong, reverting the Git change can restore the previous desired configuration. This becomes particularly useful for AI systems where changes may involve: Model version Prompt configuration Resource limits GPU requirements Inference replicas Routing policies Security should not be added after deployment. It should exist throughout the platform. A request path might look like: User ↓ Authentication ↓ API Gateway ↓ AI Application ↓ Authorized Tool / Model ↓ Protected Resource Important controls include: The model should never become the security boundary. If an AI agent requests access to a database, API, or production tool, the infrastructure still needs to verify whether that operation is allowed. A useful principle is: AI decides what it wants to do. The platform decides what it is allowed to do. This becomes especially important as AI agents begin interacting directly with operational infrastructure. AI applications may need credentials for: Model providers Databases Vector stores External APIs Cloud services MCP servers These should not appear inside: Source code Container images Git repositories Application logs Instead: Secrets Manager ↓ Workload Identity ↓ Application Where possible, workload identity is preferable to long-lived static credentials. If credentials are required, they should have: The same principle applies to development, staging, and production. Each environment should have its own trust boundary. In Part 3, we looked at AI observability in detail. At platform level, we want a common telemetry path. Applications GPU Nodes Model Servers AI Agents │ ├── Metrics ├── Logs └── Traces │ ↓ OpenTelemetry / Exporters ↓ Observability Platform OpenTelemetry provides Kubernetes tooling for collectors, operators, and workload instrumentation, making it useful as a common telemetry layer. A production AI platform should let engineers move from: User says AI is slow to: Request ID ↓ API trace ↓ Model inference latency ↓ GPU saturation ↓ Growing queue depth without searching through five unrelated systems. The platform should make diagnosis easier by default. Traditional SRE signals still matter: Availability Latency Errors Traffic AI adds another layer: Time to first token Tokens per second Queue depth GPU utilization Model errors Tool-call failures Provider latency A service might look healthy at Kubernetes level: Pods: Healthy CPU: Normal Memory: Normal while users experience: Queue: Growing TTFT: Increasing GPU: Saturated Production readiness means connecting both views. Cost should not live in a completely separate dashboard owned only by finance. The platform already knows: GPU utilization GPU hours Requests Tokens Models Tenants Workloads Those signals can be connected to cost. AI Workload ↓ Resource Usage ↓ Cost Allocation ↓ Team / Tenant / Product For example: Workload: document-summary Model: model-a GPU Hours: 420 Requests: 180,000 Cost / Request: $0.018 Now engineering teams can make better decisions about: Scaling Model choice Prompt size Caching GPU capacity FinOps becomes part of platform engineering rather than something reviewed only when the cloud bill arrives. A production AI release should move through controlled stages. Developer ↓ Pull Request ↓ Tests ↓ Security Checks ↓ Build Image ↓ Deploy to Staging ↓ Validation ↓ Production Approval ↓ GitOps Deployment Validation may include: Unit tests Integration tests Model evaluations Security tests Smoke tests Performance tests AI adds an important distinction. The service may deploy successfully while the model behaves worse. So release validation should consider both: Infrastructure Health + AI Quality A technically healthy deployment is not necessarily a successful AI release. Imagine model version v13 increases latency or produces worse results. Production should not depend on someone remembering a long sequence of commands. If configuration is version controlled: v12 ↓ v13 ↓ Problem detected ↓ Revert ↓ v12 Rollback becomes part of the deployment design. The same applies to: Container versions Prompt configurations Routing policies Resource limits Model versions Recovery should be tested before an incident occurs. Now the complete architecture starts to look like this: Developer ↓ Git Repository ↓ CI / Validation ↓ Container Registry ↓ GitOps Configuration ↓ GitOps Controller ↓ Kubernetes ┌─────────────┼─────────────┐ ↓ ↓ ↓ AI Services Model Servers AI Agents │ │ │ └─────────────┼─────────────┘ ↓ GPU Infrastructure ↓ ┌───────────────┼───────────────┐ ↓ ↓ ↓ Observability Security FinOps Supporting everything: Infrastructure as Code Identity Secrets Policies Networking Storage Testing This is less about choosing one perfect tool. It is about creating clear operational boundaries. A developer building an AI feature should not need to understand every detail of: GPU scheduling Network policy Secret rotation Prometheus configuration GitOps controllers Cloud billing Ideally, the platform provides a supported path. For example: Developer defines: Model GPU requirement Scaling policy Environment The platform handles: Infrastructure Deployment Security Observability Cost allocation This is where AI infrastructure starts overlapping with platform engineering . Modern internal developer platforms increasingly combine Kubernetes, GitOps, observability, governance, security, and self-service into standardized workflows. AI agents are now beginning to become consumers of those same platforms alongside human developers. Before calling an AI platform production-ready, I would want clear answers to these questions: If several of these depend on manual knowledge, the platform still has operational risk. This article completes the AI Infrastructure for Cloud Engineers series. We started with: Why Kubernetes? Then moved through: GPU Scheduling ↓ Model Serving ↓ Observability ↓ FinOps ↓ Production Platform The biggest lesson for me is that production AI is not only a machine-learning problem. It is also a: Cloud problem Distributed systems problem Platform engineering problem Security problem SRE problem FinOps problem And that is exactly why cloud engineers have an important role in the AI ecosystem. A model can be impressive in a notebook. A production AI system needs much more. It needs: Repeatable infrastructure Controlled delivery Secure access Reliable compute Observability Cost visibility Recovery Kubernetes provides the runtime foundation. Infrastructure as Code makes the environment reproducible. GitOps makes deployment controlled and auditable. Security defines what workloads are allowed to access. Observability tells us what the system is doing. FinOps tells us whether we are using those resources efficiently. Together, those pieces turn an AI application into an operable production platform . And for cloud, DevOps, SRE, and platform engineers, that may be one of the most interesting parts of the current AI shift. This article completes my AI Infrastructure for Cloud Engineers series: Thanks to everyone who has read, commented, or shared their experience throughout the series. I regularly share what I learn about cloud infrastructure, Kubernetes, DevOps, SRE, platform engineering, and the infrastructure behind production AI systems. LinkedIn: Connect with me on LinkedIn https://www.linkedin.com/in/sushyamnagallapati/ If you were designing an AI platform from scratch today, which part would you standardize first: infrastructure, deployment, security, observability, or cost management?